Btukfyl

I am developing Emacs package that runs from Emacs-22 to Emacs26, and I'd like to confirm that the test passes with these Emacs.

However, since each log is long, I would like to gather the return value of each test and display it clearly at the end.

How can I write this in Makefile?
The list of Emacs to be tested has been obtained in ALL_EMACS_VERS as array (22.1 23.4 24.5 25.3 26.1), and now I am running the test as follows.

EMACS          ?= emacs

LOAD_PATH      := -L $(dir $(lastword $(MAKEFILE_LIST)))

BATCH          := $(EMACS) -Q --batch $(LOAD_PATH)

ALL_EMACS_VERS := $(shell compgen -c emacs- | grep -oP '(?<=emacs-)([0-9]|.)+' | sort | uniq)



test:

    $(BATCH) -l srt-tests.el -f srt-run-tests



debug-localtest:

    for ver in $(ALL_EMACS_VERS); do 

        echo "===  test by emacs-$${ver}...  ==="; 

        EMACS=emacs-$${ver} make test; 

    done

I'd like to get information on which version the test passed, or did not pass, and output it as follows.

===  test by emacs-22.1  ===

emacs-22.1 -Q --batch -L ./ -l srt-tests.el -f srt-run-tests



Running 30 tests...

GNU Emacs 22.1.1 (mac-apple-darwin)

 of 2017-10-07 on osx339.sd.apple.com

[PASSED]  simple:equal

[PASSED]  simple:=

...



===== Run 30 Tests, 29 Expected, 1 Failed, 0 Errored =====



make[1]: Leaving directory '/Users/conao/Develop/git/.dotfiles/.emacs.d/local/26.1/site-lisp/srt.el'





===  test by emacs-23.4  ===

emacs-23.4 -Q --batch -L ./ -l srt-tests.el -f srt-run-tests



Running 30 tests...

GNU Emacs 23.4.1 (x86_64-apple-darwin, NS apple-appkit-1038.36)

 of 2012-01-29 on bob.porkrind.org

[PASSED]  simple:equal

[PASSED]  simple:=

...



===== Run 30 Tests, 29 Expected, 1 Failed, 0 Errored =====



make[1]: Leaving directory '/Users/conao/Develop/git/.dotfiles/.emacs.d/local/26.1/site-lisp/srt.el'



...

(Emacs 24.5, 25.3, 26.1)

...



===  localtest completed!!  ===



*FAILED* Emacs-22.1

*FAILED* Emacs-23.4

[PASSED] Emacs-24.5

*FAILED* Emacs-25.3

[PASSED] Emacs-26.1

The problem has two parts. The first is to give the desired output for one test. If (as the title of your Question suggests) this depends on the return value of the process, this approach will work:

single-test:

    (some-command && echo PASS) || echo FAIL

In your case, this becomes:

ARGS := -Q --batch $(LOAD_PATH) -l srt-tests.el -f srt-run-tests



test-emacs-22.1:

    (emacs-22.1 $(ARGS) && echo [PASS] emacs-22.1) || echo *FAIL* emacs-22.1

The second part is to iterate over a list of versions. I suggest using a pattern rule:

ALL_EMACS_VERS := 22.1 23.4 24.5 25.3 26.1

TESTS := $(addprefix test-emacs-, $(ALL_EMACS_VERS))



all-tests: $(TESTS)



test-%:

    ($* $(ARGS) && echo [PASS] $*) || echo *FAIL* $*

Finally, if these processes produce output you wish to suppress:

test-%:

    ($* $(ARGS) > /dev/null && echo [PASS] $*) || echo *FAIL* $*

EDIT: To print all of the results after all of the script output, the simplest approach is to save them in a file, called e.g. "results":

all-tests: $(TESTS)

    @cat results

    @rm results



test-%:

    @($* $(ARGS) && echo [PASS] $* >> results) || echo *FAIL* $* >> results