raw fixed width data

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

raw fixed width data

Wes Potts
Is it possible to configure vim such that it can distinguish records
(and treat them as lines) in a fixed-width data file (without
newlines)?

For example: if I have a data file containing a sequence of characters like:

123456789012345678901

Can I set a "record length" of say... 3, such that vim displays lines
of 3 characters:

123
456
789
012
345
678
901

I deal with very large data files (fixed width) on a regular basis.
Some have Unix EOLs, some DOS EOLs, and some have no EOLs.  I
frequently need to edit data to test programs and the data files with
no EOLs are becoming a nuisance.

Thanks,
Wes
Reply | Threaded
Open this post in threaded view
|

Re: raw fixed width data

Tim Chase-2
> For example: if I have a data file containing a sequence of
> characters like:
>
> 123456789012345678901
>
> Can I set a "record length" of say... 3, such that vim
> displays lines of 3 characters:
>
> 123
> 456
> 789

If you don't mind actually editing your data (or a copy thereof),
its something I've done before with simply

        :%s/.\{3}/&\r/g

which will insert a newline after every third character.  Adjust
accordingly for your favorite record-length :)

If the files are huge and slow Vim down, you can do something
similar with sed, if you've got access to it:

        sed 's/.\{3\}/&\n/g' infile.txt > outfile.txt

> I frequently need to edit data to test programs and the data
> files with no EOLs are becoming a nuisance.

So the solution is to put EOLs in where you want 'em :)

Hope this helps,

-tim





Reply | Threaded
Open this post in threaded view
|

Re: raw fixed width data

Wes Potts
On 8/2/05, Tim Chase <[hidden email]> wrote:

> > For example: if I have a data file containing a sequence of
> > characters like:
> >
> > 123456789012345678901
> >
> > Can I set a "record length" of say... 3, such that vim
> > displays lines of 3 characters:
> >
> > 123
> > 456
> > 789
>
> If you don't mind actually editing your data (or a copy thereof),
> its something I've done before with simply
>
>         :%s/.\{3}/&\r/g
>
> which will insert a newline after every third character.  Adjust
> accordingly for your favorite record-length :)
>
> If the files are huge and slow Vim down, you can do something
> similar with sed, if you've got access to it:
>
>         sed 's/.\{3\}/&\n/g' infile.txt > outfile.txt

I've added newlines with perl before but, I'm hoping to find a
"better" solution as these files are sometimes HUGE (Gigabytes) and
when making edits repeatedly, a 4 or 5 step process ( copy, add EOLs,
edit, remove EOLs, copy back ) can be tedious.  So, I'd prefer a
solution that would allow "in-place" editing.  What I'm looking for is
a "record" editor.

>
> > I frequently need to edit data to test programs and the data
> > files with no EOLs are becoming a nuisance.
>
> So the solution is to put EOLs in where you want 'em :)
>
> Hope this helps,
>
> -tim
>

Thanks for the input.

Wes
Reply | Threaded
Open this post in threaded view
|

Re: raw fixed width data

jose isaias cabrera-2

If you have programmed before, I would suggest to write a small d program
for it.

http://www.digitalmars.com/

D is a like a combination of c, c++, java and c#, but better and faster.
:-) It's a compiled language, so...  The string capabilities are amazing.
You could write your own record editor or creator.  I have changed my
programming language to D.

Just a thought...


----- Original Message -----
From: "Wes Potts" <[hidden email]>
To: "Tim Chase" <[hidden email]>
Cc: <[hidden email]>
Sent: Tuesday, August 02, 2005 4:52 PM
Subject: Re: raw fixed width data


On 8/2/05, Tim Chase <[hidden email]> wrote:

> > For example: if I have a data file containing a sequence of
> > characters like:
> >
> > 123456789012345678901
> >
> > Can I set a "record length" of say... 3, such that vim
> > displays lines of 3 characters:
> >
> > 123
> > 456
> > 789
>
> If you don't mind actually editing your data (or a copy thereof),
> its something I've done before with simply
>
>         :%s/.\{3}/&\r/g
>
> which will insert a newline after every third character.  Adjust
> accordingly for your favorite record-length :)
>
> If the files are huge and slow Vim down, you can do something
> similar with sed, if you've got access to it:
>
>         sed 's/.\{3\}/&\n/g' infile.txt > outfile.txt

I've added newlines with perl before but, I'm hoping to find a
"better" solution as these files are sometimes HUGE (Gigabytes) and
when making edits repeatedly, a 4 or 5 step process ( copy, add EOLs,
edit, remove EOLs, copy back ) can be tedious.  So, I'd prefer a
solution that would allow "in-place" editing.  What I'm looking for is
a "record" editor.

>
> > I frequently need to edit data to test programs and the data
> > files with no EOLs are becoming a nuisance.
>
> So the solution is to put EOLs in where you want 'em :)
>
> Hope this helps,
>
> -tim
>

Thanks for the input.

Wes

Reply | Threaded
Open this post in threaded view
|

Re: raw fixed width data

A.J.Mechelynck
In reply to this post by Wes Potts
----- Original Message -----
From: "Wes Potts" <[hidden email]>
To: <[hidden email]>
Sent: Tuesday, August 02, 2005 10:15 PM
Subject: raw fixed width data


> Is it possible to configure vim such that it can distinguish records
> (and treat them as lines) in a fixed-width data file (without
> newlines)?
>
> For example: if I have a data file containing a sequence of characters
> like:
>
> 123456789012345678901
>
> Can I set a "record length" of say... 3, such that vim displays lines
> of 3 characters:
>
> 123
> 456
> 789
> 012
> 345
> 678
> 901
>
> I deal with very large data files (fixed width) on a regular basis.
> Some have Unix EOLs, some DOS EOLs, and some have no EOLs.  I
> frequently need to edit data to test programs and the data files with
> no EOLs are becoming a nuisance.
>
> Thanks,
> Wes
>
>
>

WARNING: the following is untested.

At start of edit, let's say you want a fixed length of 50 characters:

    :%s/\.{50}/\0\r/g

This will break lines after every 50 characters not including ends-of-lines.

At end of edit (if required)

    :%join!

This removes all ends-of-lines.


HTH,
Tony.


Reply | Threaded
Open this post in threaded view
|

Re: raw fixed width data

Tofer Chagnon
In reply to this post by Wes Potts
On 8/2/05, Wes Potts <[hidden email]> wrote:

> Is it possible to configure vim such that it can distinguish records
> (and treat them as lines) in a fixed-width data file (without
> newlines)?
>
> For example: if I have a data file containing a sequence of characters like:
>
> 123456789012345678901
>
> Can I set a "record length" of say... 3, such that vim displays lines
> of 3 characters:
>
> 123
> 456
> 789
> 012
> 345
> 678
> 901
>

You could try doing a vertical split, and then adjusting the window
width so that the desired number of columns show.  You might want to
do :nnoremap j gj :nnoremap k gk for this editing style.

Tofer
Reply | Threaded
Open this post in threaded view
|

Re: raw fixed width data

Tim Chase-2
In reply to this post by Wes Potts
>>        sed 's/.\{3\}/&\n/g' infile.txt > outfile.txt
>
> I've added newlines with perl before but, I'm hoping to find a
> "better" solution as these files are sometimes HUGE (Gigabytes) and
> when making edits repeatedly, a 4 or 5 step process ( copy, add EOLs,
> edit, remove EOLs, copy back ) can be tedious.  So, I'd prefer a
> solution that would allow "in-place" editing.  What I'm looking for is
> a "record" editor.

Ah...I didn't gather the frustration of the multiple steps to
begin with :)

Vim does allow you to automate some of this, as you can see in the

        :help hex-editing

This does presume you have access to "xxd".  You may be able to
tweak it to use "xxd -c 3" to spit out 3-column format.  however,
I don't know whether it requires you to edit both the binary and
the visual goods.

Alternatively, if you've got your perl script, or the previous
sed command, or even the previous vim command, you can do the
same sort of exercise using autocommands:

        augroup Binary
          au!
          au BufReadPost  *.bin %s/.\{3}/&\r/g
          au BufWritePre *.bin %j!
          au BufWritePost *.bin set nomod
        augroup END

Seems to do the trick for me, and in a slightly more automated
fashion than hand cranking it.

Hope this helps (more),  :)

-tim



Reply | Threaded
Open this post in threaded view
|

Re: raw fixed width data

Marian Csontos
In reply to this post by Wes Potts
On Tue, 02 Aug 2005 22:52:40 +0200, Wes Potts <[hidden email]> wrote:

> On 8/2/05, Tim Chase <[hidden email]> wrote:
>> > For example: if I have a data file containing a sequence of
>> > characters like:
>> >
>> > 123456789012345678901
>> >
>> > Can I set a "record length" of say... 3, such that vim
>> > displays lines of 3 characters:
>> >
>> > 123
>> > 456
>> > 789
>>
>> If you don't mind actually editing your data (or a copy thereof),
>> its something I've done before with simply
>>
>>         :%s/.\{3}/&\r/g
>>
>> which will insert a newline after every third character.  Adjust
>> accordingly for your favorite record-length :)
>>
>> If the files are huge and slow Vim down, you can do something
>> similar with sed, if you've got access to it:
>>
>>         sed 's/.\{3\}/&\n/g' infile.txt > outfile.txt
>
> I've added newlines with perl before but, I'm hoping to find a
> "better" solution as these files are sometimes HUGE (Gigabytes) and
> when making edits repeatedly, a 4 or 5 step process ( copy, add EOLs,
> edit, remove EOLs, copy back ) can be tedious.  So, I'd prefer a
> solution that would allow "in-place" editing.  What I'm looking for is
> a "record" editor.
>

Try
:set wrap
:set columns=<record-width>

You can't move using standard keys - up, down arrows, jk, ^, 0, $, home,  
end,... - instead you have to use gj, gk, g^, g0, g$,...
But You can map them:
nmap <down> gj
imap <down> <C-O>gj
nmap <up>   gk
imap <up>   <C-O>gk
...

Regards

-- Marian


________ Information from NOD32 ________
This message was checked by NOD32 Antivirus System for Linux Mail Server.
http://www.nod32.com
Reply | Threaded
Open this post in threaded view
|

Re: raw fixed width data

Steve Vance
In reply to this post by Wes Potts
--- In [hidden email], Tim Chase <vim@t...> wrote:
 >     augroup Binary
 >       au!
 >       au BufReadPost  *.bin %s/.\{3}/&\r/g
 >       au BufWritePre *.bin %j!

Careful, the %j! will also get rid of newlines which were in the file to
begin with.
Maybe something better like
     %s/\(.\{3}\)\n/\1/g
but I'm not sure if that does it, either.

   stv
Reply | Threaded
Open this post in threaded view
|

Re: raw fixed width data

Paul-433
In reply to this post by Wes Potts
On Tue, 2 Aug 2005, Wes Potts wrote:

> solution that would allow "in-place" editing.  What I'm looking for is
> a "record" editor.

I have no idea if this would be any use, but it may be just the thing for you:

http://record-editor.sourceforge.net/

--

.