Skip to content

Commit 2bfa1f6

Browse files
rossantrossant
committed
Updating recipe 5.11.
1 parent c4e349a commit 2bfa1f6

File tree

1 file changed

+167
-221
lines changed

1 file changed

+167
-221
lines changed

‎notebooks/chapter05_hpc/11_mpi.ipynb‎

Lines changed: 167 additions & 221 deletions
Original file line numberDiff line numberDiff line change
@@ -1,222 +1,168 @@
1-
{
2-
"metadata": {
3-
"name": ""
4-
},
5-
"nbformat": 3,
6-
"nbformat_minor": 0,
7-
"worksheets": [
8-
{
9-
"cells": [
10-
{
11-
"cell_type": "markdown",
12-
"metadata": {
13-
"word_id": "4818_05_mpi"
14-
},
15-
"source": [
16-
"# Using MPI with IPython\n",
17-
"\n",
18-
"**Message Passing Interface (MPI)** is a standardized communication protocol for parallel systems. It is used in many parallel computing applications to exchange data between nodes. MPI has a high barrier to entry, but it is very efficient and powerful.\n",
19-
"\n",
20-
"IPython's parallel computing system has been designed from the ground up to work with MPI. If you are new to MPI, it is a good idea to start using it with IPython. If you are an experienced MPI user, you will find that IPython integrates seamlessly with your parallel application.\n",
21-
"\n",
22-
"In this recipe, we show how to use MPI with IPython through a very simple example."
23-
]
24-
},
25-
{
26-
"cell_type": "markdown",
27-
"metadata": {},
28-
"source": [
29-
"## Getting started\n",
30-
"\n",
31-
"To use MPI with IPython you need:\n",
32-
"\n",
33-
"* A standard MPI implementation such as [OpenMPI](http://www.open-mpi.org) or [MPICH](http://www.mpich.org).\n",
34-
"* The [mpi4py package](http://mpi4py.scipy.org).\n",
35-
"\n",
36-
"For example, here are the commands to install MPI for IPython on Ubuntu:\n",
37-
"\n",
38-
" > sudo apt-get install libcr-dev mpich2 mpich2-doc\n",
39-
" > sudo apt-get install python-mpi4py\n",
40-
" "
41-
]
42-
},
43-
{
44-
"cell_type": "markdown",
45-
"metadata": {},
46-
"source": [
47-
"## How to do it..."
48-
]
49-
},
50-
{
51-
"cell_type": "markdown",
52-
"metadata": {},
53-
"source": [
54-
"1. We first need to create a MPI profile with:"
55-
]
56-
},
57-
{
58-
"cell_type": "code",
59-
"collapsed": false,
60-
"input": [
61-
"!ipython profile create --parallel --profile=mpi"
62-
],
63-
"language": "python",
64-
"metadata": {},
65-
"outputs": [],
66-
"prompt_number": 1
67-
},
68-
{
69-
"cell_type": "markdown",
70-
"metadata": {},
71-
"source": [
72-
"2. Then, we need to open `~/.ipython/profile_mpi/ipcluster_config.py` and add the line `c.IPClusterEngines.engine_launcher_class = 'MPI'`."
73-
]
74-
},
75-
{
76-
"cell_type": "markdown",
77-
"metadata": {},
78-
"source": [
79-
"3. Once the MPI profile has been created and configured, we can launch the engines in the IPython dashboard, by selecting the number of engines (e.g. one per processor) in the *Clusters* tab, *MPI* profile, and pressing *Start*. Alternatively, we can run in a terminal: `ipcluster start -n 2 --engines MPI --profile=mpi`."
80-
]
81-
},
82-
{
83-
"cell_type": "markdown",
84-
"metadata": {},
85-
"source": [
86-
"4. Now, to actually use the engines, we create a MPI client in the notebook."
87-
]
88-
},
89-
{
90-
"cell_type": "code",
91-
"collapsed": false,
92-
"input": [
93-
"import numpy as np\n",
94-
"from IPython.parallel import Client"
95-
],
96-
"language": "python",
97-
"metadata": {},
98-
"outputs": [],
99-
"prompt_number": 2
100-
},
101-
{
102-
"cell_type": "code",
103-
"collapsed": false,
104-
"input": [
105-
"c = Client(profile='mpi')"
106-
],
107-
"language": "python",
108-
"metadata": {},
109-
"outputs": [],
110-
"prompt_number": 3
111-
},
112-
{
113-
"cell_type": "markdown",
114-
"metadata": {},
115-
"source": [
116-
"5. Let's create a view on all engines."
117-
]
118-
},
119-
{
120-
"cell_type": "code",
121-
"collapsed": false,
122-
"input": [
123-
"view = c[:]"
124-
],
125-
"language": "python",
126-
"metadata": {},
127-
"outputs": [],
128-
"prompt_number": 4
129-
},
130-
{
131-
"cell_type": "markdown",
132-
"metadata": {},
133-
"source": [
134-
"6. In this example, we compute the sum of all integers between 0 and 15 in parallel over two cores. We first distribute the array with the 16 values across the engines (each engine gets a subarray)."
135-
]
136-
},
137-
{
138-
"cell_type": "code",
139-
"collapsed": false,
140-
"input": [
141-
"view.scatter('a', np.arange(16., dtype='float'))"
142-
],
143-
"language": "python",
144-
"metadata": {},
145-
"outputs": [
146-
{
147-
"metadata": {},
148-
"output_type": "pyout",
149-
"prompt_number": 4,
150-
"text": [
151-
"<AsyncResult: scatter>"
152-
]
153-
}
154-
],
155-
"prompt_number": 5
156-
},
157-
{
158-
"cell_type": "markdown",
159-
"metadata": {},
160-
"source": [
161-
"7. We compute the total sum in parallel using MPI's `allreduce` function. Every node makes the same computation and returns the same result."
162-
]
163-
},
164-
{
165-
"cell_type": "code",
166-
"collapsed": false,
167-
"input": [
168-
"%%px\n",
169-
"from mpi4py import MPI\n",
170-
"import numpy as np\n",
171-
"print MPI.COMM_WORLD.allreduce(np.sum(a), op=MPI.SUM)"
172-
],
173-
"language": "python",
174-
"metadata": {},
175-
"outputs": [
176-
{
177-
"output_type": "stream",
178-
"stream": "stdout",
179-
"text": [
180-
"[stdout:0] 120.0\n",
181-
"[stdout:1] 120.0\n"
182-
]
183-
}
184-
],
185-
"prompt_number": 6
186-
},
187-
{
188-
"cell_type": "markdown",
189-
"metadata": {},
190-
"source": [
191-
"## How it works...\n",
192-
"\n",
193-
"In this example, each node:\n",
194-
"\n",
195-
"1. receives a subset of the integers,\n",
196-
"2. computes the local sum of those integers,\n",
197-
"3. sends this local sum to all other engines,\n",
198-
"4. receives the local sum of the other engines,\n",
199-
"5. computes the total sum of those local sums.\n",
200-
"\n",
201-
"This is how **all-reduce** works in MPI: the principle is to **scatter** data across engines first, then to **reduce** the local computations through a global operator (here, `MPI.SUM`).\n",
202-
"\n",
203-
"There are many other parallel computing paradigms in MPI. You can find more information here:\n",
204-
"\n",
205-
"* [MPI tutorials by Wes Kendall](http://mpitutorial.com)\n",
206-
"* [MPI tutorials by Blaise Barney, Lawrence Livermore National Laboratory](https://computing.llnl.gov/tutorials/mpi/)"
207-
]
208-
},
209-
{
210-
"cell_type": "markdown",
211-
"metadata": {},
212-
"source": [
213-
"## See also\n",
214-
"\n",
215-
"* Distribute your code across multiple cores with IPython"
216-
]
217-
}
218-
],
219-
"metadata": {}
220-
}
221-
]
1+
{
2+
"metadata": {
3+
"name": "",
4+
"signature": "sha256:36700138892ccc6e580b3f5bf11affd6a83785f7665f1e211b823818f0f69653"
5+
},
6+
"nbformat": 3,
7+
"nbformat_minor": 0,
8+
"worksheets": [
9+
{
10+
"cells": [
11+
{
12+
"cell_type": "markdown",
13+
"metadata": {},
14+
"source": [
15+
"> This is one of the 100 recipes of the [IPython Cookbook](http://ipython-books.github.io/), the definitive guide to high-performance scientific computing and data science in Python."
16+
]
17+
},
18+
{
19+
"cell_type": "markdown",
20+
"metadata": {
21+
"word_id": "4818_05_mpi"
22+
},
23+
"source": [
24+
"# 5.11. Using MPI with IPython"
25+
]
26+
},
27+
{
28+
"cell_type": "markdown",
29+
"metadata": {},
30+
"source": [
31+
"To use MPI with IPython you need:\n",
32+
"\n",
33+
"* A standard MPI implementation such as [OpenMPI](http://www.open-mpi.org) or [MPICH](http://www.mpich.org).\n",
34+
"* The [mpi4py package](http://mpi4py.scipy.org).\n",
35+
"\n",
36+
"For example, here are the commands to install MPI for IPython on Ubuntu:\n",
37+
"\n",
38+
" > sudo apt-get install libcr-dev mpich2 mpich2-doc\n",
39+
" > sudo apt-get install python-mpi4py"
40+
]
41+
},
42+
{
43+
"cell_type": "markdown",
44+
"metadata": {},
45+
"source": [
46+
"1. We first need to create a MPI profile with:"
47+
]
48+
},
49+
{
50+
"cell_type": "code",
51+
"collapsed": false,
52+
"input": [
53+
"!ipython profile create --parallel --profile=mpi"
54+
],
55+
"language": "python",
56+
"metadata": {},
57+
"outputs": []
58+
},
59+
{
60+
"cell_type": "markdown",
61+
"metadata": {},
62+
"source": [
63+
"2. Then, we need to open `~/.ipython/profile_mpi/ipcluster_config.py` and add the line `c.IPClusterEngines.engine_launcher_class = 'MPI'`."
64+
]
65+
},
66+
{
67+
"cell_type": "markdown",
68+
"metadata": {},
69+
"source": [
70+
"3. Once the MPI profile has been created and configured, we can launch the engines in the IPython dashboard, by selecting the number of engines (e.g. one per processor) in the *Clusters* tab, *MPI* profile, and pressing *Start*. Alternatively, we can run in a terminal: `ipcluster start -n 2 --engines MPI --profile=mpi`."
71+
]
72+
},
73+
{
74+
"cell_type": "markdown",
75+
"metadata": {},
76+
"source": [
77+
"4. Now, to actually use the engines, we create a MPI client in the notebook."
78+
]
79+
},
80+
{
81+
"cell_type": "code",
82+
"collapsed": false,
83+
"input": [
84+
"import numpy as np\n",
85+
"from IPython.parallel import Client"
86+
],
87+
"language": "python",
88+
"metadata": {},
89+
"outputs": []
90+
},
91+
{
92+
"cell_type": "code",
93+
"collapsed": false,
94+
"input": [
95+
"c = Client(profile='mpi')"
96+
],
97+
"language": "python",
98+
"metadata": {},
99+
"outputs": []
100+
},
101+
{
102+
"cell_type": "markdown",
103+
"metadata": {},
104+
"source": [
105+
"5. Let's create a view on all engines."
106+
]
107+
},
108+
{
109+
"cell_type": "code",
110+
"collapsed": false,
111+
"input": [
112+
"view = c[:]"
113+
],
114+
"language": "python",
115+
"metadata": {},
116+
"outputs": []
117+
},
118+
{
119+
"cell_type": "markdown",
120+
"metadata": {},
121+
"source": [
122+
"6. In this example, we compute the sum of all integers between 0 and 15 in parallel over two cores. We first distribute the array with the 16 values across the engines (each engine gets a subarray)."
123+
]
124+
},
125+
{
126+
"cell_type": "code",
127+
"collapsed": false,
128+
"input": [
129+
"view.scatter('a', np.arange(16., dtype='float'))"
130+
],
131+
"language": "python",
132+
"metadata": {},
133+
"outputs": []
134+
},
135+
{
136+
"cell_type": "markdown",
137+
"metadata": {},
138+
"source": [
139+
"7. We compute the total sum in parallel using MPI's `allreduce` function. Every node makes the same computation and returns the same result."
140+
]
141+
},
142+
{
143+
"cell_type": "code",
144+
"collapsed": false,
145+
"input": [
146+
"%%px\n",
147+
"from mpi4py import MPI\n",
148+
"import numpy as np\n",
149+
"print MPI.COMM_WORLD.allreduce(np.sum(a), op=MPI.SUM)"
150+
],
151+
"language": "python",
152+
"metadata": {},
153+
"outputs": []
154+
},
155+
{
156+
"cell_type": "markdown",
157+
"metadata": {},
158+
"source": [
159+
"> You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).\n",
160+
"\n",
161+
"> [IPython Cookbook](http://ipython-books.github.io/), by [Cyrille Rossant](http://cyrille.rossant.net), Packt Publishing, 2014 (400 pages). [Get a 50% discount by pre-ordering now](http://www.packtpub.com/ipython-interactive-computing-and-visualization-cookbook/book) with the code `mK00gPxQM` (time-limited offer)!"
162+
]
163+
}
164+
],
165+
"metadata": {}
166+
}
167+
]
222168
}

0 commit comments

Comments
 (0)