make test enhancement

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

make test enhancement

Petr Splichal
Hi all!

Recently I've noticed that the Vim's "make test" does not return
appropriate exit code if something goes wrong during the testing.

This makes the usual sequence "make && make test && make install"
unusable. Including such a test inside a source package makes no
sense and actually does not prevent any possible failures which
should be it's main purpose I guess.

The following comment from the Makefile is not true, as the "ALL
DONE" message appears always, regardless of the actual test result:
# If everything is alright, the final message will be "ALL DONE".

Moreover, the final result output of the test seems quite brief to
myself. From the report it's not clear which/how many tests were
actually run:

    Test results:
    ALL DONE

I've created a simple patch that fixes all the issues mentioned.
What do you think about it? Thank you for your time! And special
thanks to Bram for the Vim itself! It's really a great tool! :-)

psss...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

FAIL before:

    # make test
    ...
    test3 FAILED
    ALL DONE
    # echo $?
    0

FAIL after:

    # make test
    ...
    Test results:
    test1 PASS
    test2 PASS
    test3 FAILED
    test4 PASS
    test5 PASS
    test6 PASS
    ...
    SOME TESTS FAILED
    make: *** [nongui] Error 1
    # echo $?
    2
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---


make-test.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

make test enhancement

Petr Splichal
Hi all!

Recently I've noticed that the Vim's "make test" does not return
appropriate exit code if something goes wrong during the testing.

This makes the usual sequence "make && make test && make install"
unusable. Including such a test inside a source package makes no
sense and actually does not prevent any possible failures which
should be it's main purpose I guess.

The following comment from the Makefile is not true, as the "ALL
DONE" message appears always, regardless of the actual test result:
# If everything is alright, the final message will be "ALL DONE".

Moreover, the final result output of the test seems quite brief to
myself. From the report it's not clear which/how many tests were
actually run:

    Test results:
    ALL DONE

I've created a simple patch that fixes all the issues mentioned.
What do you think about it? Thank you for your time! And special
thanks to Bram for the Vim itself! It's really a great tool! :-)

psss...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

FAIL before:

    # make test
    ...
    test3 FAILED
    ALL DONE
    # echo $?
    0

FAIL after:

    # make test
    ...
    Test results:
    test1 PASS
    test2 PASS
    test3 FAILED
    test4 PASS
    test5 PASS
    test6 PASS
    ...
    SOME TESTS FAILED
    make: *** [nongui] Error 1
    # echo $?
    2
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---


make-test.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: make test enhancement

Bram Moolenaar


Petr Splichal wrote:

> Recently I've noticed that the Vim's "make test" does not return
> appropriate exit code if something goes wrong during the testing.
>
> This makes the usual sequence "make && make test && make install"
> unusable. Including such a test inside a source package makes no
> sense and actually does not prevent any possible failures which
> should be it's main purpose I guess.
>
> The following comment from the Makefile is not true, as the "ALL
> DONE" message appears always, regardless of the actual test result:
> # If everything is alright, the final message will be "ALL DONE".
>
> Moreover, the final result output of the test seems quite brief to
> myself. From the report it's not clear which/how many tests were
> actually run:
>
>     Test results:
>     ALL DONE
>
> I've created a simple patch that fixes all the issues mentioned.
> What do you think about it? Thank you for your time! And special
> thanks to Bram for the Vim itself! It's really a great tool! :-)
>
> psss...
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> FAIL before:
>
>     # make test
>     ...
>     test3 FAILED
>     ALL DONE
>     # echo $?
>     0
>
> FAIL after:
>
>     # make test
>     ...
>     Test results:
>     test1 PASS
>     test2 PASS
>     test3 FAILED
>     test4 PASS
>     test5 PASS
>     test6 PASS
>     ...
>     SOME TESTS FAILED
>     make: *** [nongui] Error 1
>     # echo $?
>     2

Looks good to me.  Objections anyone?

--
Engineers are widely recognized as superior marriage material: intelligent,
dependable, employed, honest, and handy around the house.
                                (Scott Adams - The Dilbert principle)

 /// Bram Moolenaar -- [hidden email] -- http://www.Moolenaar.net   \\\
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\        download, build and distribute -- http://www.A-A-P.org        ///
 \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

--
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
Reply | Threaded
Open this post in threaded view
|

Re: make test enhancement

Lech Lorens
In reply to this post by Petr Splichal
On 25-Nov-2009 Petr Splichal <[hidden email]> wrote:
> Hi all!
>
> Recently I've noticed that the Vim's "make test" does not return
> appropriate exit code if something goes wrong during the testing.
>
> This makes the usual sequence "make && make test && make install"
> unusable.

[...]

Actually... the comment in src/Makefile is perfectly valid. The problem
is that you seem to have worked on src/testdir/Makefile as it was in Vim
7.2.000, while in 7.2.015 it was changed so that "ALL DONE" is only
displayed in case of all tests having passed and "make && make test &&
make install" works the way you wanted it to.

There still remains the issue of displaying which tests have been
performed, though, which - I believe - is a matter of personal taste.
I have no opinion here.

To prevent problems with outdated files from occurring in the future you
might want to use one of the repositories tracking Vim source code:
- SVN (described here: http://www.vim.org/subversion.php)
- CVS (described here: http://www.vim.org/cvs.php)
- git (available here: http://repo.or.cz/w/vim_mainline.git)

There is also the vim-extended git repository, which additionally holds
code for a few Vim extensions: http://repo.or.cz/w/vim_extended.git

--
Cheers,
Lech

--
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
Reply | Threaded
Open this post in threaded view
|

Re: make test enhancement

Petr Splichal
In reply to this post by Petr Splichal
> Actually... the comment in src/Makefile is perfectly valid. The problem
> is that you seem to have worked on src/testdir/Makefile as it was in Vim
> 7.2.000, while in 7.2.015 it was changed so that "ALL DONE" is only
> displayed in case of all tests having passed and "make && make test &&
> make install" works the way you wanted it to.

Oh, I've just downloaded the latest tar archive from the web
before. Now I know where are the most fresh bits. It's nice to
find out this has already been fixed.

> There still remains the issue of displaying which tests have been
> performed, though, which - I believe - is a matter of personal taste.
> I have no opinion here.

As the output from the "make test" is really fast and contains a
lot of known & expected errors & warning messages which can be
confusing at the first sight (happened to me) --- I think adding a
nice summary at the end won't hurt and will make it absolutely
clear that everything went all right.

As you said, this is rather matter of style & personal taste and
not that important as the right exit code return. Anyway, if you
decide to include this a little bit more verbose behaviour I'm
attaching a simple patch for that.

> To prevent problems with outdated files from occurring in the future you
> might want to use one of the repositories tracking Vim source code:
> - SVN (described here: http://www.vim.org/subversion.php)
> - CVS (described here: http://www.vim.org/cvs.php)
> - git (available here: http://repo.or.cz/w/vim_mainline.git)

Thanks for the links! ;-)

psss...

--
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php

make-test-updated.patch (1K) Download Attachment