Quantcast

Can vim view VERY large files w/o having to load entire file into memory?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Can vim view VERY large files w/o having to load entire file into memory?

txporter

We currently use an old version of the crisp editor (running in read-
only mode) on our solaris boxes to view extremely large files, in some
cases approaching 2G.  The nice thing about crisp is that it will pull
a small chunk of the file into memory and immediately display it, then
as you scroll through the file, it will pull discard the first chunk
and load another.  You see the first screenful of data immediately, no
matter what the file size, and you can limit the amount of memory that
crisp uses without affecting the size of file you can load.

We are switching to mostly linux boxes and have not found a way to run
vim in a similar mode:  using vim -R file causes vim to apparently
freeze until either the file is loaded into memory, or you hit Ctrl-C
to interrupt the load.  I do not know how to interrupt this load
behavior if I try to open two files at the same time using vim -R -o
file1 file2.  I believe there is a way to limit the amount of memory
vim uses on startup, but have not hit the magic command options for
this yet.

less is not an answer because our files are often wider than 80 chars
and we require horizontal as well as vertical scrolling.

We currently have a kludge wrapper that does a head -10000 on the file
to a temp file and them lets us look at the results using vim, but in
the cases where data being sought is not in the first 10000 records
this does not help.  I know a scripter/unix geek approach would be to
grep for the desired data in such a large file and look at the
results, but many of our users are not so unix savvy, and prefer the
simple read-only file browser with arrow key navigation that crisp
gives us.

Any sugggestions?

Thanks,

Tom Porter

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_use" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can vim view VERY large files w/o having to load entire file into memory?

Dominique Pellé

Tom Porter wrote:

> We currently use an old version of the crisp editor (running in read-
> only mode) on our solaris boxes to view extremely large files, in some
> cases approaching 2G.  The nice thing about crisp is that it will pull
> a small chunk of the file into memory and immediately display it, then
> as you scroll through the file, it will pull discard the first chunk
> and load another.  You see the first screenful of data immediately, no
> matter what the file size, and you can limit the amount of memory that
> crisp uses without affecting the size of file you can load.

...snip...

I would personally also enjoy something like that. It's something that
had already been discussed, at least in this thread:

http://www.mail-archive.com/vim-dev@.../msg03165.html

Being able to open a file quickly would be great.  Rather than the
extreme solution of loading pages of the file in memory,  I would not
mind if the file got loaded entirely in memory, as along as it's done
in a lazy way, i.e. pressing G to go to the end for example could
cause to wait until the file is entirely loaded (fine, doesn't "less" do
just that?). But browsing the top of the file could become
instantaneous. Perceived time when opening the file (to show the
top of the file) can be more important to users than actual time to
load the entire file.

Unfortunately, it's probably not so simple to implement in Vim,
and there are other priorities.

It probably also opens lots of questions. For example,
say I have this in my ~/.vimrc...

  set fileencodings=ucs-bom,utf-8,latin1

.... so that Vim selects the encoding based on file content.
The top of the file might be valid-utf-8, but near the end, there
might be invalid utf-8 sequences.  If Vim loads the file lazily,
it can't figure it out, so it can't select the encoding automatically
until it has read the entire file. It could be a documented limitation
when turning a lazy file loading option. But I'm sure there are
many other issues like this.

I don't think this idea even appears on the already huge todo list
anyway. I searched in ":help todo.txt" and could not see anything
like that.

In the mean time, there are settings which can help to speed
up Vim significantly when using large files.  See:

http://vim.wikia.com/wiki/VimTip343

Regards
-- Dominique

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_use" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can vim view VERY large files w/o having to load entire file into memory?

Robert R. Melton
In reply to this post by txporter

On 4/3/2009 10:49 PM, Tom Porter wrote:
> [...]
> We currently have a kludge wrapper that does a head -10000 on the file
> to a temp file and them lets us look at the results using vim, but in
> the cases where data being sought is not in the first 10000 records
> this does not help.  I know a scripter/unix geek approach would be to
> grep for the desired data in such a large file and look at the
> results, but many of our users are not so unix savvy, and prefer the
> simple read-only file browser with arrow key navigation that crisp
> gives us.

I am a bit startled by this ... your users are currently looking thru
10,000+ records using arrow keys?  That would, at first glance seem
insane.  I fear I must be missing some critical part of the work-flow.
Does Crisp feature a search interface they are using to do that job that
grep would do?

I know it is slightly off-topic, but now I am really curious...

--
Robert Melton | http://robertmelton.com/contact

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_use" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can vim view VERY large files w/o having to load entire file into memory?

Tony Mechelynck
In reply to this post by Dominique Pellé

On 04/04/09 05:44, Dominique Pelle wrote:

>
> Tom Porter wrote:
>
>> We currently use an old version of the crisp editor (running in read-
>> only mode) on our solaris boxes to view extremely large files, in some
>> cases approaching 2G.  The nice thing about crisp is that it will pull
>> a small chunk of the file into memory and immediately display it, then
>> as you scroll through the file, it will pull discard the first chunk
>> and load another.  You see the first screenful of data immediately, no
>> matter what the file size, and you can limit the amount of memory that
>> crisp uses without affecting the size of file you can load.
>
> ...snip...
>
> I would personally also enjoy something like that. It's something that
> had already been discussed, at least in this thread:
>
> http://www.mail-archive.com/vim-dev@.../msg03165.html
>
> Being able to open a file quickly would be great.  Rather than the
> extreme solution of loading pages of the file in memory,  I would not
> mind if the file got loaded entirely in memory, as along as it's done
> in a lazy way, i.e. pressing G to go to the end for example could
> cause to wait until the file is entirely loaded (fine, doesn't "less" do
> just that?). But browsing the top of the file could become
> instantaneous. Perceived time when opening the file (to show the
> top of the file) can be more important to users than actual time to
> load the entire file.
>
> Unfortunately, it's probably not so simple to implement in Vim,
> and there are other priorities.
>
> It probably also opens lots of questions. For example,
> say I have this in my ~/.vimrc...
>
>    set fileencodings=ucs-bom,utf-8,latin1

a rather common setting if you're a "Westerner" and your 'encoding' is
set to UTF-8. I have the same. People from East Asia might have their
favourite CJK encoding between utf-8 and latin1 but the reasoning below
still applies, maybe even more strongly.

>
> .... so that Vim selects the encoding based on file content.
> The top of the file might be valid-utf-8, but near the end, there
> might be invalid utf-8 sequences.  If Vim loads the file lazily,
> it can't figure it out, so it can't select the encoding automatically
> until it has read the entire file. It could be a documented limitation
> when turning a lazy file loading option. But I'm sure there are
> many other issues like this.
>
> I don't think this idea even appears on the already huge todo list
> anyway. I searched in ":help todo.txt" and could not see anything
> like that.
>
> In the mean time, there are settings which can help to speed
> up Vim significantly when using large files.  See:
>
> http://vim.wikia.com/wiki/VimTip343
>
> Regards
> -- Dominique

...not to mention modelines, which (if you set 'modeline' on and leave
'modelines' at its default) can be either in the first five or the last
five lines of the file, so both ends have to be examined. OK, it might
be possible to go to the 6th-last linefeed without reading everything
that goes before it, but I'm not sure how to do it buglessly, and the
capability to go to offset X from the end of the file without even
reading what comes before could depend on the OS and/or on the
filesystem type. And for multibyte encoding detection (see above) it's
the _whole_ file which has to be examined in order to _keep_ a multibyte
value like utf-8 anyway. (To _reject_ it, finding one invalid sequence
is enough.)


Best regards,
Tony.
--
Our policy is, when in doubt, do the right thing.
                -- Roy L. Ash, ex-president Litton Industries

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_use" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can vim view VERY large files w/o having to load entire file into memory?

Tony Mechelynck
In reply to this post by Robert R. Melton

On 04/04/09 05:56, Robert R. Melton wrote:

>
> On 4/3/2009 10:49 PM, Tom Porter wrote:
>> [...]
>> We currently have a kludge wrapper that does a head -10000 on the file
>> to a temp file and them lets us look at the results using vim, but in
>> the cases where data being sought is not in the first 10000 records
>> this does not help.  I know a scripter/unix geek approach would be to
>> grep for the desired data in such a large file and look at the
>> results, but many of our users are not so unix savvy, and prefer the
>> simple read-only file browser with arrow key navigation that crisp
>> gives us.
>
> I am a bit startled by this ... your users are currently looking thru
> 10,000+ records using arrow keys?  That would, at first glance seem
> insane.  I fear I must be missing some critical part of the work-flow.
> Does Crisp feature a search interface they are using to do that job that
> grep would do?
>
> I know it is slightly off-topic, but now I am really curious...
>

Not only arrow keys. Don't forget the search capability, and commands
such as (normal) 1234G or (ex-command) :1234 to go directly to a known
line number. Also 75% to go to a given percentage of the file's length.

I have one very large file, the Unihan database, a copy of a text file
from the Unicode site, each line of which includes three tab-separated
fields, namely, Unicode codepoint, key name (there are quite a number of
possible keys, and usually several are present on successive lines for
each codepoint) and key value. You could equate it to as many
Dictionaries (in the Vim sense) as there are Unified CJK Characters.
Quite bulky. Well, I never go very far into it with just up-and-down
arrowing (let's say to different keys for a given codepoint, at most two
or rarely three screen heights) but I use regexp searching quite a lot,
either by codepoint or by key+value: for instance, I could search for
codepoints whose meaning includes the word "big", for characters made
from Radical 140 "grass" plus five additional strokes, for words which
can be read er4 in Mandarin or futatsu in Japanese-Kun, and so on and so
forth.


Best regards,
Tony.
--
DENNIS: You can't expect to wield supreme executive power just 'cause some
         watery tart threw a sword at you!
                  "Monty Python and the Holy Grail" PYTHON (MONTY)
PICTURES LTD

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_use" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can vim view VERY large files w/o having to load entire file into memory?

James Freer-2

Interesting you should mention this editing of large files. I used to
work in the oil industry and was involved with vast text files of data
and the only editor that could cope with it was a little freebie
called editor.com [dos]. I've never seen it since although i still
have a copy if you wanted to try it. It would seem that sometimes the
smaller apps are worth using.

Have you tried the other smaller editors like joe and such like... may
surprise a bit like this editor.com did for me. It seems to me that
the top featured editors don't consider vast data files within their
design.

james

--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_use" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can vim view VERY large files w/o having to load entire file into memory?

txporter
In reply to this post by Robert R. Melton



On Apr 3, 11:56 pm, "Robert R. Melton" <[hidden email]> wrote:

> On 4/3/2009 10:49 PM, Tom Porter wrote:
>
> > [...]
> > We currently have a kludge wrapper that does a head -10000 on the file
> > to a temp file and them lets us look at the results using vim, but in
> > the cases where data being sought is not in the first 10000 records
> > this does not help.  I know a scripter/unix geek approach would be to
> > grep for the desired data in such a large file and look at the
> > results, but many of our users are not so unix savvy, and prefer the
> > simple read-only file browser with arrow key navigation that crisp
> > gives us.
>

They arrow-key through primarily to scan location of fields, then
search for id numbers using typical vim '/' search.

> I am a bit startled by this ... your users are currently looking thru
> 10,000+ records using arrow keys?  That would, at first glance seem
> insane.  I fear I must be missing some critical part of the work-flow.
> Does Crisp feature a search interface they are using to do that job that
> grep would do?
>
> I know it is slightly off-topic, but now I am really curious...
>
> --
> Robert Melton |http://robertmelton.com/contact
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_use" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can vim view VERY large files w/o having to load entire file into memory?

John Little-4
In reply to this post by txporter

> less is not an answer because our files are often wider than 80 chars
> and we require horizontal as well as vertical scrolling.

Are you sure?  I thought recent versions of less support sideways
scrolling, less -S.

Regards, John
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_use" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---

Loading...