Skip to content

Conversation

@jaggzh
Copy link

@jaggzh jaggzh commented Dec 4, 2018

Summary

Added an option to model.summary() for a bit more brief output.
Adding short=True will now just disable the underlines between layers.
The output might be unclear on complex models, but it's up to the caller.
Example outputs with normal (default) summary, and with short=True:

Default model.summary() (eg. short=False):

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
Model (InputLayer)           (None, 112, 24, 1)        0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 110, 22, 40)       400
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 54, 10, 48)        17328
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 26, 4, 96)         41568
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 26, 4, 96)         9312
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 12, 1, 192)        166080
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 5, 1, 384)         221568
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 2, 1, 768)         885504
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 2, 1, 384)         295296
_________________________________________________________________
flatten_1 (Flatten)          (None, 768)               0
_________________________________________________________________
dense_1 (Dense)              (None, 1024)              787456
_________________________________________________________________
dense_2 (Dense)              (None, 1024)              1049600
_________________________________________________________________
dense_3 (Dense)              (None, 2)                 2050
=================================================================
Total params: 3,476,162
Trainable params: 3,476,162
Non-trainable params: 0
_________________________________________________________________
None

With model.summary(short=True):

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
Model (InputLayer)           (None, 112, 24, 1)        0
conv2d_1 (Conv2D)            (None, 110, 22, 40)       400
conv2d_2 (Conv2D)            (None, 54, 10, 48)        17328
conv2d_3 (Conv2D)            (None, 26, 4, 96)         41568
conv2d_4 (Conv2D)            (None, 26, 4, 96)         9312
conv2d_5 (Conv2D)            (None, 12, 1, 192)        166080
conv2d_6 (Conv2D)            (None, 5, 1, 384)         221568
conv2d_7 (Conv2D)            (None, 2, 1, 768)         885504
conv2d_8 (Conv2D)            (None, 2, 1, 384)         295296
flatten_1 (Flatten)          (None, 768)               0
dense_1 (Dense)              (None, 1024)              787456
dense_2 (Dense)              (None, 1024)              1049600
dense_3 (Dense)              (None, 2)                 2050
=================================================================
Total params: 3,476,162
Trainable params: 3,476,162
Non-trainable params: 0
_________________________________________________________________
None

Related Issues

PR Overview

  • This PR requires new unit tests [y/n] (make sure tests are included)
  • This PR requires to update the documentation [y/n] (make sure the docs are up-to-date)
  • This PR is backwards compatible [y/n]
  • This PR changes the current API [y/n] (all API changes need to be approved by fchollet)

Long term update of local repo
@akbargumbira
Copy link

I am not sure if it's just me, but I think this is better and could just replace the previous one?

@jaggzh
Copy link
Author

jaggzh commented Dec 5, 2018

Seems clear enough with more complex layer inputs/outputs too...
(I chopped out some columns and layers just to make it easier to see in the post -- nothing that affects the vertical divisions though):

_________________________________________________________________________________________                 
Layer (type)                    Output Shape       Param #  Connected to                                  
=========================================================================================                 
noise_input (InputLayer)        (None, 100)        0                                                      
auxilary_input (InputLayer)     (None, 47)         0                                                      
concatenate_1 (Concatenate)     (None, 147)        0        noise_input[0][0]                             
                                                            auxilary_input[0][0]                          
dense_1 (Dense)                 (None, 1000)       148000   concatenate_1[0][0]                           
batch_normalization_1 (BatchNor (None, 1000)       4000     dense_1[0][0]                                 
leaky_re_lu_1 (LeakyReLU)       (None, 1000)       0        batch_normalization_1[0][0]                   
...                                                                                                       
batch_normalization_4 (BatchNor (None, 400, 1)     4        conv1d_3[0][0]                                
activation_1 (Activation)       (None, 400, 1)     0        batch_normalization_4[0][0]                   
=========================================================================================

@farizrahman4u
Copy link
Contributor

I don't like the short argument. Pick one :)

@jaggzh
Copy link
Author

jaggzh commented Dec 7, 2018

I updated the commit to just remove the underlines between layers. No short= option now. The people have spoken.

@jaggzh jaggzh closed this Dec 7, 2018
@jaggzh jaggzh reopened this Dec 7, 2018
@farizrahman4u
Copy link
Contributor

Need @fchollet review since UX related.

@gabrieldemarmiesse
Copy link
Contributor

I like the new style :)

@farizrahman4u
Copy link
Contributor

I like it too.

@jaggzh
Copy link
Author

jaggzh commented Dec 15, 2018 via email

@gabrieldemarmiesse
Copy link
Contributor

I don't think it's related to your PR. Please wait for #11521 to be merged into master. This should fix the timeout.

@fchollet
Copy link
Collaborator

Let's go with the new style, but I think we can improve it further. I think we need column separators, especially useful when columns are overlapping. Let's try to add that?

@gabrieldemarmiesse
Copy link
Contributor

If we add delimiters, it will look like this:

__________________________________________________________________________________________           
Layer (type)                   | Output Shape     | Param # |Connected to                                 
===============================|==================|=========|=============================                
noise_input (InputLayer)       | (None, 100)      | 0       |                                              
auxilary_input (InputLayer)    | (None, 47)       | 0       |                                               
concatenate_1 (Concatenate)    | (None, 147)      | 0       | noise_input[0][0]                             
                               |                  |         | auxilary_input[0][0]                          
dense_1 (Dense)                | (None, 1000)     | 148000  | concatenate_1[0][0]                           
batch_normalization_1 (BatchNor| (None, 1000)     | 4000    | dense_1[0][0]                                 
leaky_re_lu_1 (LeakyReLU)      | (None, 1000)     | 0       | batch_normalization_1[0][0]                   
...                            |                  |         |                                                
batch_normalization_4 (BatchNor| (None, 400, 1)   | 4       | conv1d_3[0][0]                                
activation_1 (Activation)      | (None, 400, 1)   | 0       | batch_normalization_4[0][0]                   
=========================================================================================

It can easily become markdown compatible:

Layer (type)                   | Output Shape     | Param # |Connected to                                 
-------------------------------|------------------|---------|-----------------------------                
noise_input (InputLayer)       | (None, 100)      | 0       |                                              
auxilary_input (InputLayer)    | (None, 47)       | 0       |                                               
concatenate_1 (Concatenate)    | (None, 147)      | 0       | noise_input[0][0]                             
...                            |                  |         | auxilary_input[0][0]                          
dense_1 (Dense)                | (None, 1000)     | 148000  | concatenate_1[0][0]                           
batch_normalization_1 (BatchNor| (None, 1000)     | 4000    | dense_1[0][0]                                 
leaky_re_lu_1 (LeakyReLU)      | (None, 1000)     | 0       | batch_normalization_1[0][0]                   
...                            |                  |         |                                                
batch_normalization_4 (BatchNor| (None, 400, 1)   | 4       | conv1d_3[0][0]                                
activation_1 (Activation)      | (None, 400, 1)   | 0       | batch_normalization_4[0][0]                   

Which becomes:

Layer (type) Output Shape Param # Connected to
noise_input (InputLayer) (None, 100) 0
auxilary_input (InputLayer) (None, 47) 0
concatenate_1 (Concatenate) (None, 147) 0 noise_input[0][0]
... auxilary_input[0][0]
dense_1 (Dense) (None, 1000) 148000 concatenate_1[0][0]
batch_normalization_1 (BatchNor (None, 1000) 4000 dense_1[0][0]
leaky_re_lu_1 (LeakyReLU) (None, 1000) 0 batch_normalization_1[0][0]
...
batch_normalization_4 (BatchNor (None, 400, 1) 4 conv1d_3[0][0]
activation_1 (Activation) (None, 400, 1) 0 batch_normalization_4[0][0]

The text version isn't as nice, but we get markdown for free. What do you think?

@fchollet
Copy link
Collaborator

I'm worried about what happens with multi-input layers, where there are multiple lines needed for the rightmost column. Could you show an example?

I think 99.9% of usage will be in terminals or notebooks, so we should optimize for that use case only. People who want markdown can convert the previous output to markdown in just a couple of text editor commands.

@gabrieldemarmiesse
Copy link
Contributor

Indeed, compatibility is going to make a lot of things harder. The gain is small but not enough to justify the complexity increase.

@jaggzh
Copy link
Author

jaggzh commented Dec 24, 2018

  1. My jupyter notebooks just output the summary() as text -- not caring about its format. Is there another issue with that that I'm unaware of?
  2. The main issue with Markdown is the issue of blank fields (cells) -- it chokes up on them. The solution for that is to either: 1. Preface line with a pipe (see example below), or 2. Put a placeholder in blank fields (a dash maybe) (no example shown, but you can imagine it).
  3. Multiple inputs seem okay to me; tell me what you think below.
  4. Lastly, the pipe at the beginning of the line actually makes the post-model summary look nicer, because I had to make the last === divider line into fields so it wouldn't screw up the table.
  5. I'm not encouraging anything here -- I just wanted the shorter summary. :) That being said, maybe a markdown-compatible one would facilitate online clean-looking user interactions with markdown compatible sites; HOWEVER, their table implementation might vary.

In any case, here is a sample with multiple inputs, and a sub-model (a convnet) used twice. It seems to produce a nice markdown table, and the text isn't ugly, but it does have the '|''s prepended.

| Layer (type)                   | Output Shape       | Param #   | Connected to
| ------------------------------ | ------------------ | --------- | ------------------------------
| in_0 (InputLayer)              | (None, 50, 50, 1)  | 0         |
| in_1 (InputLayer)              | (None, 50, 50, 1)  | 0         |
| model_1 (Model)                | (None, 400)        | 19100157  | in_0[0][0]
|                                |                    |           | in_1[0][0]
| inps_joined (Concatenate)      | (None, 800)        | 0         | model_1[1][0]
|                                |                    |           | model_1[2][0]
| dense_1 (Dense)                | (None, 128)        | 102528    | inps_joined[0][0]
| dense_2 (Dense)                | (None, 128)        | 102528    | inps_joined[0][0]
| activation_13 (Activation)     | (None, 128)        | 0         | dense_1[0][0]
| activation_14 (Activation)     | (None, 128)        | 0         | dense_2[0][0]
| fl_0 (Dense)                   | (None, 1)          | 129       | activation_13[0][0]
| fl_1 (Dense)                   | (None, 1)          | 129       | activation_14[0][0]
| ============================== | ================== | ========= | ==============================
Total params: 19,305,471
Trainable params: 19,293,753
Non-trainable params: 11,718
Layer (type) Output Shape Param # Connected to
in_0 (InputLayer) (None, 50, 50, 1) 0
in_1 (InputLayer) (None, 50, 50, 1) 0
model_1 (Model) (None, 400) 19100157 in_0[0][0]
in_1[0][0]
inps_joined (Concatenate) (None, 800) 0 model_1[1][0]
model_1[2][0]
dense_1 (Dense) (None, 128) 102528 inps_joined[0][0]
dense_2 (Dense) (None, 128) 102528 inps_joined[0][0]
activation_13 (Activation) (None, 128) 0 dense_1[0][0]
activation_14 (Activation) (None, 128) 0 dense_2[0][0]
fl_0 (Dense) (None, 1) 129 activation_13[0][0]
fl_1 (Dense) (None, 1) 129 activation_14[0][0]
============================== ================== ========= ==============================
Total params: 19,305,471
Trainable params: 19,293,753
Non-trainable params: 11,718

@jaggzh
Copy link
Author

jaggzh commented Dec 24, 2018

Here it is without the prefixed pipes, and with underscores to preserve otherwise blank fields:

Layer (type)                   | Output Shape       | Param #   | Connected to
------------------------------ | ------------------ | --------- | ------------------------------  
in_0 (InputLayer)              | (None, 50, 50, 1)  | 0         | _
in_1 (InputLayer)              | (None, 50, 50, 1)  | 0         | _
model_1 (Model)                | (None, 400)        | 19100157  | in_0[0][0]
_                              | _                  | _         | in_1[0][0]
inps_joined (Concatenate)      | (None, 800)        | 0         | model_1[1][0]
_                              | _                  | _         | model_1[2][0]
dense_1 (Dense)                | (None, 128)        | 102528    | inps_joined[0][0]
dense_2 (Dense)                | (None, 128)        | 102528    | inps_joined[0][0]
activation_13 (Activation)     | (None, 128)        | 0         | dense_1[0][0]
activation_14 (Activation)     | (None, 128)        | 0         | dense_2[0][0]
fl_0 (Dense)                   | (None, 1)          | 129       | activation_13[0][0]
fl_1 (Dense)                   | (None, 1)          | 129       | activation_14[0][0]

Total params: 19,305,471
Trainable params: 19,293,753
Non-trainable params: 11,718
Layer (type) Output Shape Param # Connected to
in_0 (InputLayer) (None, 50, 50, 1) 0 _
in_1 (InputLayer) (None, 50, 50, 1) 0 _
model_1 (Model) (None, 400) 19100157 in_0[0][0]
_ _ _ in_1[0][0]
inps_joined (Concatenate) (None, 800) 0 model_1[1][0]
_ _ _ model_1[2][0]
dense_1 (Dense) (None, 128) 102528 inps_joined[0][0]
dense_2 (Dense) (None, 128) 102528 inps_joined[0][0]
activation_13 (Activation) (None, 128) 0 dense_1[0][0]
activation_14 (Activation) (None, 128) 0 dense_2[0][0]
fl_0 (Dense) (None, 1) 129 activation_13[0][0]
fl_1 (Dense) (None, 1) 129 activation_14[0][0]

Total params: 19,305,471
Trainable params: 19,293,753
Non-trainable params: 11,718

@fchollet
Copy link
Collaborator

fchollet commented Jan 1, 2019

The version from here is the best so far, so let's go with that #11795 (comment)

Is there anything we can do to make it easier to read the last column when there are layers that have multiple inputs?

@MrinalJain17
Copy link

The version from here is the best so far, so let's go with that #11795 (comment)

Is there anything we can do to make it easier to read the last column when there are layers that have multiple inputs?

Maybe adding blank rows only around layers with multiple inputs could work. It's a bit more readable in my opinion.

@fchollet @gabrieldemarmiesse @jaggzh

Layer (type)                   | Output Shape       | Param #   | Connected to
------------------------------ | ------------------ | --------- | ------------------------------  
in_0 (InputLayer)              | (None, 50, 50, 1)  | 0         | _
in_1 (InputLayer)              | (None, 50, 50, 1)  | 0         | _
                               |                    |           |
model_1 (Model)                | (None, 400)        | 19100157  | in_0[0][0]
_                              | _                  | _         | in_1[0][0]
                               |                    |           |
inps_joined (Concatenate)      | (None, 800)        | 0         | model_1[1][0]
_                              | _                  | _         | model_1[2][0]
                               |                    |           |
dense_1 (Dense)                | (None, 128)        | 102528    | inps_joined[0][0]
dense_2 (Dense)                | (None, 128)        | 102528    | inps_joined[0][0]
activation_13 (Activation)     | (None, 128)        | 0         | dense_1[0][0]
activation_14 (Activation)     | (None, 128)        | 0         | dense_2[0][0]
fl_0 (Dense)                   | (None, 1)          | 129       | activation_13[0][0]
fl_1 (Dense)                   | (None, 1)          | 129       | activation_14[0][0]

Total params: 19,305,471
Trainable params: 19,293,753
Non-trainable params: 11,718

However, I am not sure of the complexity that such an implementation might induce.

@gabrieldemarmiesse
Copy link
Contributor

I propose another way of handling layers with multiple inputs:

Layer (type)                   | Output Shape       | Param #   | Connected to
------------------------------ | ------------------ | --------- | ------------------------------  
in_0 (InputLayer)              | (None, 50, 50, 1)  | 0         | _
in_1 (InputLayer)              | (None, 50, 50, 1)  | 0         | _
model_1 (Model)                | (None, 400)        | 19100157  | [in_0[0][0], in_1[0][0]]
inps_joined (Concatenate)      | (None, 800)        | 0         | [model_1[1][0], model_1[2][0]]
dense_1 (Dense)                | (None, 128)        | 102528    | inps_joined[0][0]
dense_2 (Dense)                | (None, 128)        | 102528    | inps_joined[0][0]
activation_13 (Activation)     | (None, 128)        | 0         | dense_1[0][0]
activation_14 (Activation)     | (None, 128)        | 0         | dense_2[0][0]
fl_0 (Dense)                   | (None, 1)          | 129       | activation_13[0][0]
fl_1 (Dense)                   | (None, 1)          | 129       | activation_14[0][0]

Total params: 19,305,471
Trainable params: 19,293,753
Non-trainable params: 11,718

quite pytonic and close to how the layer is called on the tensors. Opinions?

@gabrieldemarmiesse gabrieldemarmiesse changed the title Added an option to model.summary() for a bit more brief output. Making the output of model.summary() easier to read. Jan 10, 2019
@farizrahman4u
Copy link
Contributor

@gabrieldemarmiesse that looks neat. Also, what about changing "Connected to" to "Inputs"?

@gabrieldemarmiesse
Copy link
Contributor

@farizrahman4u yes that makes way more sense since we are displaying tensors, not layers.

@fchollet
Copy link
Collaborator

Also, what about changing "Connected to" to "Inputs"?

+1

The use of [] looks good to me, thanks @gabrieldemarmiesse.

In terms of visual style, I think this #11795 (comment) is the best looking one. If we make sure we cut bleeding text (like batch_normalization_1 (BatchNor) early enough, there will be spaces between the context of columns and we won't need vertical delimiters. The style of horizontal delimiter in the example I linked also looks better.

@jaggzh
Copy link
Author

jaggzh commented Jan 11, 2019

@MrinalJain17 The blank lines don't look as bad as one might expect. Thanks for that. I think, though, that we can balance minimal-size, visual appeal, and consistency, though.

@fchollet My personal tendency is still to provide options to users in some way. They might load up some arbitrary model and print the summary(), so a nice default, but then the user can set an option:
Markdown-table compatible, CSV, force-full-fields (ie. no truncation of batch_norma).

In any case. I made a sample default below. Notes:

  1. (I made column 1 short-width by hand, to chop the layer name/type for demonstration)
  2. Column 1 is an exception, truncated so it has a minimum of 3 spaces at the end for clarity.
  3. @gabrieldemarmiesse For normal terminal output, the width might quickly cause a wrap, destroying the clarity. I used your idea, though, with a reversion to the all-whitespace format desired by @fchollet . Technically, the inputs are now copy-and-paste'able almost directly now too. :)
  4. Column name is now "Inputs"
  5. If we can have some keras.options [for output format] in the future, it might be convenient (csv? full-width)? The following format is visually-appealing, but actually cumbersome to convert to markdown by a user (and get it right, with the pipes, empty-cell fillers, removal of ==== lines that break the markdown recognition, splitting of the other divider lines into cells, etc.). NEVERTHELESS, they can just paste this as pre-formatted text -- the markdown doesn't confer much advantage (are they going to paste it in a spreadsheet?), so...

...what do you think about this for the current version (and default if we offer options in the future, or separate calls, like model.summary_csv()). But for now:

________________________________________________________________________________
Layer (type)               Output Shape       Param #   Inputs
================================================================================
in_0 (InputLayer)          (None, 50, 50, 1)  0         
in_1 (InputLayer)          (None, 50, 50, 1)  0         
model_1 (Model)            (None, 400)        19100157  [in_0[0][0],
                                                         in_1[0][0]]
inps_joined (Concatenate   (None, 800)        0         [model_1[1][0],
                                                         model_1[2][0]]
dense_1 (Dense)            (None, 128)        102528    inps_joined[0][0]
dense_2 (Dense)            (None, 128)        102528    inps_joined[0][0]
activation_13 (Activatio   (None, 128)        0         dense_1[0][0]
activation_14 (Activatio   (None, 128)        0         dense_2[0][0]
fl_0 (Dense)               (None, 1)          129       activation_13[0][0]
fl_1 (Dense)               (None, 1)          129       activation_14[0][0]
================================================================================
Total params: 19,305,471
Trainable params: 19,293,753
Non-trainable params: 11,718

@gabrieldemarmiesse
Copy link
Contributor

If we can have some keras.options [for output format] in the future, it might be convenient (csv? full-width)? The following format is visually-appealing, but actually cumbersome to convert to markdown by a user (and get it right, with the pipes, empty-cell fillers, removal of ==== lines that break the markdown recognition, splitting of the other divider lines into cells, etc.). NEVERTHELESS, they can just paste this as pre-formatted text -- the markdown doesn't confer much advantage (are they going to paste it in a spreadsheet?), so...

I would vote against adding options for those reasons:

  • It's not clear if a significant number of users would use those options.
  • We have to support those options, so maintenance burden increases.
  • It complexifies the API.
  • If users want a computer-readable text to understand the model structure, they can do model.to_json()
  • It is very easy for users or an external package to write their own custom ``model.summary()`.

For normal terminal output, the width might quickly cause a wrap, destroying the clarity. I used your idea, though, with a reversion to the all-whitespace format desired by @fchollet . Technically, the inputs are now copy-and-paste'able almost directly now too. :)

We have a line_lenght argument. See https://keras.io/utils/#print_summary . If the text fits, it might be preferable to put everything in the same line. Otherwise, we can use your solution and use a blank line.

+1 for the separators (vertical and horizontal) they look nice.

@fchollet
Copy link
Collaborator

The new example looks great, thanks a lot. We can go with this! I agree with @gabrieldemarmiesse that we shouldn't add more formatting options at this time.

wassname added a commit to wassname/torchsummaryX that referenced this pull request Jul 7, 2019
…ng units

This shows a trainable parameter in the full data frame and trainable vs non-trainable params in the summary. This should be like keras [example](keras-team/keras#11795 (comment)).

I also changed the pandas display so that instead of putting (M) or (K) manually in the label, pandas appends it to the number automatically.
@PhilipMay
Copy link
Contributor

The new example looks great, thanks a lot. We can go with this! I agree with @gabrieldemarmiesse that we shouldn't add more formatting options at this time.

So should this PR be closed then?

@jaggzh
Copy link
Author

jaggzh commented Aug 26, 2019 via email

@fchollet
Copy link
Collaborator

Thank you for preparing this PR.

We are no longer adding new features to multi-backend Keras, as we are refocusing development efforts on tf.keras. If you are still interested in submitting this PR, please direct it to tf.keras in the TensorFlow repository instead.

I think improving the UX of summary is valuable and worth pursuing in tf.keras.

@fchollet fchollet closed this Sep 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants