Btukfyl

I have several big files with some measurements.

It looks this way:

N 12344;PE 9.9999999;...

#S 0 0 31 44 75 130 165 196...

#S_+ "2 5 2 3 3 1 1 2 3 1 2 2...



N 12345;PE 9.9999999;...

#S 0 0 34 57 84 133 152...

#S_+ "1 0 1 1 2 3 0 0 0...



N 12346;PE 9.9999999;...

#S 0 0 31 44 73 140 169...

#S_+ "3 3 4 0 0 2 1 2 4...



N 25104;PE 9.9999999;...

#S 0 0 36 52 102 108 145...

#S_+ "1 1 0 1 0 0 3 0 1...



N 25105;PE 9.9999999;...

#S 0 0 32 58 88 130 143...

Sample is here:
http://pasted.co/d9806b7c4

The file is much bigger but I replaced part of the data with "..." to make it shorter.

I need to somehow replace the line ends before "#S" - in fact simply merge the "N" line with the following two ones into one line (or with the following three ones so I can get rid of the blank lines). Expect output like this:

N 12344;PE 9.9999999; #S 0 0 31 44 75 130 165 196 #S_+ "2 5 2 3 3 1 1 2 3 1 2 2...

N 12345;PE 9.9999999; #S 0 0 34 57 84 133 152 #S_+ "1 0 1 1 2 3 0 0 0...

N 12346;PE 9.9999999; #S 0 0 31 44 73 140 169 #S_+ "3 3 4 0 0 2 1 2 4...

N 25104;PE 9.9999999; #S 0 0 36 52 102 108 145 #S_+ "1 1 0 1 0 0 3 0 1...

N 25105;PE 9.9999999; #S 0 0 32 58 88 130 143...

Is this possible to achieve using some command-line utility in linux?

My knowledge is quite limited in this area so I would appreciate any help.

thanks

With sed:

sed -z -e 's/n#S/ #S/g' -e 's/nN /N /g' data

In slow-mo:

• 
  -z makes sed consider the file as a single line (so the line ends are plain characters)


• 
  's/n#S/#S/g' replaces all LF's occurring just before a #S by a space


• 
  -e 's/nN /N /g' replaces all LFs before N (ie, the blank lines)

This is a portable solution with POSIX sed, implementing the following rules:

• empty lines shall be deleted;


• any line starting with #S shall be merged with the previous non-empty line, with a single space character between them, unless there is no previous non-empty line.

The code:

<data sed '/^$/ d; :start; N; s/n$//; t start; s/n#S/ #S/; t start; P; D'

The same with comments (still working code):

<data sed '

  /^$/ d      # If empty line read, delete it and start a new cycle.

  :start      # A label.

  N           # Read additional line, there are now two lines in the pattern space.

  s/n$//     # If the second line is empty, replace the newline with nothing.

  t start     # If the above replacement occurred, go to start (to add another line).

              # Otherwise

  s/n#S/ #S/ # if the second line starts with #S, replace the newline with space.

  t start     # If the above replacement occurred, go to start (to add another line).

              # Otherwise

              # (i.e when non-empty line not starting with #S occurred)

  P           # print the pattern space up to the first newline and...

  D           # delete the initial segment of the pattern space

              # through the first newline (i.e. everything just printed),

              # and start the next cycle with the resultant pattern space

              # and without reading any new input

              # (in our case the new input will be explicitly read by N then).

  '

With paste (this requires to always have groups of 4 lines):

 paste -s -d '   n' data

In slo-mo:

• 
  paste -s concatenates the lines from the file


• 
  -d specifies characters to be inserted as delimiters. When there are several characters, they are used in a round-robin fashion, so with 3 spaces and a LF:
  
  
  
  
  • the first space is used on the first splice (N to #S),
  
  
  • the second space is used on the second splice (#S to #S),
  
  
  • the third space is used on the thrid splice (#S to blank line),
  
  
  • the last delimiter, a LF, is used on the fourth splice (blank line to N)
  
  
  • and the cycle repeats for the next 4 lines.
  
  
  
  

Using Perl:

perl -0 -ape 's/R(?=RN|#)/ /g' file.txt

N 12344;PE 9.9999999;... #S 0 0 31 44 75 130 165 196... #S_+ "2 5 2 3 3 1 1 2 3 1 2 2...

N 12345;PE 9.9999999;... #S 0 0 34 57 84 133 152... #S_+ "1 0 1 1 2 3 0 0 0...

N 12346;PE 9.9999999;... #S 0 0 31 44 73 140 169... #S_+ "3 3 4 0 0 2 1 2 4...

N 25104;PE 9.9999999;... #S 0 0 36 52 102 108 145... #S_+ "1 1 0 1 0 0 3 0 1...

N 25105;PE 9.9999999;... #S 0 0 32 58 88 130 143...

Regex explain:

s/              : substitute

    R          : any kind of line break (ie. r, n, rn)

    (?=         : positive lookahead, zero-length assertion that make sure we have after

        RN     : a line break followed by letter N

      |         : OR

        #       : # character

    )           : end lookahead

/ /g            : replace with a space, global

awk (gawk)

As usually other than sed you can use awk (and in many different ways...)

awk 'ORS=" "; NR % 4 == 0 && ORS="n" ' data

where

• 
  ORS=" " fixes the output record separator, by default a newline, to a space (you can change)


• 
  NR % 4 == 0 && ORS="n" each 4th line it fixes back to the newline n
  


• If nothing else is specified awk prints the full line


• 
  data is your data file.

If you want you can use regex as in sed (in a similar way).

You can do it with any text editor that support regular expressions like Notepad++.

The new line is just simple non-printable character or two characters. In Windows usually CarrigeReturn and LineFeed and in Unix based system usually LineFeed only.

To see them you need to turn on showing non-printable character (usually a Paragraph icon)
See here: https://imgur.com/cqiTvrp

Now what you need to do is to use regular expression replacer (CTRL + H) to replace CRLF#S to #S.
The symbol for CR is r and for LF is n. So you gonna end up with rn#S or n#S to #S.
https://imgur.com/GoeVn70

Or you can replace it to SPACE if you need.