diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..169bb6f --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +.Gemfile.lock +Gemfile.lock +_site/ diff --git a/.whitesource b/.whitesource new file mode 100644 index 0000000..d7eebc0 --- /dev/null +++ b/.whitesource @@ -0,0 +1,3 @@ +{ + "settingsInheritedFrom": "VividCortex/whitesource-config@master" +} \ No newline at end of file diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..2e72ca1 --- /dev/null +++ b/LICENSE @@ -0,0 +1,427 @@ +Attribution-ShareAlike 4.0 International + +======================================================================= + +Creative Commons Corporation ("Creative Commons") is not a law firm and +does not provide legal services or legal advice. Distribution of +Creative Commons public licenses does not create a lawyer-client or +other relationship. Creative Commons makes its licenses and related +information available on an "as-is" basis. Creative Commons gives no +warranties regarding its licenses, any material licensed under their +terms and conditions, or any related information. Creative Commons +disclaims all liability for damages resulting from their use to the +fullest extent possible. + +Using Creative Commons Public Licenses + +Creative Commons public licenses provide a standard set of terms and +conditions that creators and other rights holders may use to share +original works of authorship and other material subject to copyright +and certain other rights specified in the public license below. The +following considerations are for informational purposes only, are not +exhaustive, and do not form part of our licenses. + + Considerations for licensors: Our public licenses are + intended for use by those authorized to give the public + permission to use material in ways otherwise restricted by + copyright and certain other rights. Our licenses are + irrevocable. Licensors should read and understand the terms + and conditions of the license they choose before applying it. + Licensors should also secure all rights necessary before + applying our licenses so that the public can reuse the + material as expected. Licensors should clearly mark any + material not subject to the license. This includes other CC- + licensed material, or material used under an exception or + limitation to copyright. More considerations for licensors: + wiki.creativecommons.org/Considerations_for_licensors + + Considerations for the public: By using one of our public + licenses, a licensor grants the public permission to use the + licensed material under specified terms and conditions. If + the licensor's permission is not necessary for any reason--for + example, because of any applicable exception or limitation to + copyright--then that use is not regulated by the license. Our + licenses grant only permissions under copyright and certain + other rights that a licensor has authority to grant. Use of + the licensed material may still be restricted for other + reasons, including because others have copyright or other + rights in the material. A licensor may make special requests, + such as asking that all changes be marked or described. + Although not required by our licenses, you are encouraged to + respect those requests where reasonable. More_considerations + for the public: + wiki.creativecommons.org/Considerations_for_licensees + +======================================================================= + +Creative Commons Attribution-ShareAlike 4.0 International Public +License + +By exercising the Licensed Rights (defined below), You accept and agree +to be bound by the terms and conditions of this Creative Commons +Attribution-ShareAlike 4.0 International Public License ("Public +License"). To the extent this Public License may be interpreted as a +contract, You are granted the Licensed Rights in consideration of Your +acceptance of these terms and conditions, and the Licensor grants You +such rights in consideration of benefits the Licensor receives from +making the Licensed Material available under these terms and +conditions. + + +Section 1 -- Definitions. + + a. Adapted Material means material subject to Copyright and Similar + Rights that is derived from or based upon the Licensed Material + and in which the Licensed Material is translated, altered, + arranged, transformed, or otherwise modified in a manner requiring + permission under the Copyright and Similar Rights held by the + Licensor. For purposes of this Public License, where the Licensed + Material is a musical work, performance, or sound recording, + Adapted Material is always produced where the Licensed Material is + synched in timed relation with a moving image. + + b. Adapter's License means the license You apply to Your Copyright + and Similar Rights in Your contributions to Adapted Material in + accordance with the terms and conditions of this Public License. + + c. BY-SA Compatible License means a license listed at + creativecommons.org/compatiblelicenses, approved by Creative + Commons as essentially the equivalent of this Public License. + + d. Copyright and Similar Rights means copyright and/or similar rights + closely related to copyright including, without limitation, + performance, broadcast, sound recording, and Sui Generis Database + Rights, without regard to how the rights are labeled or + categorized. For purposes of this Public License, the rights + specified in Section 2(b)(1)-(2) are not Copyright and Similar + Rights. + + e. Effective Technological Measures means those measures that, in the + absence of proper authority, may not be circumvented under laws + fulfilling obligations under Article 11 of the WIPO Copyright + Treaty adopted on December 20, 1996, and/or similar international + agreements. + + f. Exceptions and Limitations means fair use, fair dealing, and/or + any other exception or limitation to Copyright and Similar Rights + that applies to Your use of the Licensed Material. + + g. License Elements means the license attributes listed in the name + of a Creative Commons Public License. The License Elements of this + Public License are Attribution and ShareAlike. + + h. Licensed Material means the artistic or literary work, database, + or other material to which the Licensor applied this Public + License. + + i. Licensed Rights means the rights granted to You subject to the + terms and conditions of this Public License, which are limited to + all Copyright and Similar Rights that apply to Your use of the + Licensed Material and that the Licensor has authority to license. + + j. Licensor means the individual(s) or entity(ies) granting rights + under this Public License. + + k. Share means to provide material to the public by any means or + process that requires permission under the Licensed Rights, such + as reproduction, public display, public performance, distribution, + dissemination, communication, or importation, and to make material + available to the public including in ways that members of the + public may access the material from a place and at a time + individually chosen by them. + + l. Sui Generis Database Rights means rights other than copyright + resulting from Directive 96/9/EC of the European Parliament and of + the Council of 11 March 1996 on the legal protection of databases, + as amended and/or succeeded, as well as other essentially + equivalent rights anywhere in the world. + + m. You means the individual or entity exercising the Licensed Rights + under this Public License. Your has a corresponding meaning. + + +Section 2 -- Scope. + + a. License grant. + + 1. Subject to the terms and conditions of this Public License, + the Licensor hereby grants You a worldwide, royalty-free, + non-sublicensable, non-exclusive, irrevocable license to + exercise the Licensed Rights in the Licensed Material to: + + a. reproduce and Share the Licensed Material, in whole or + in part; and + + b. produce, reproduce, and Share Adapted Material. + + 2. Exceptions and Limitations. For the avoidance of doubt, where + Exceptions and Limitations apply to Your use, this Public + License does not apply, and You do not need to comply with + its terms and conditions. + + 3. Term. The term of this Public License is specified in Section + 6(a). + + 4. Media and formats; technical modifications allowed. The + Licensor authorizes You to exercise the Licensed Rights in + all media and formats whether now known or hereafter created, + and to make technical modifications necessary to do so. The + Licensor waives and/or agrees not to assert any right or + authority to forbid You from making technical modifications + necessary to exercise the Licensed Rights, including + technical modifications necessary to circumvent Effective + Technological Measures. For purposes of this Public License, + simply making modifications authorized by this Section 2(a) + (4) never produces Adapted Material. + + 5. Downstream recipients. + + a. Offer from the Licensor -- Licensed Material. Every + recipient of the Licensed Material automatically + receives an offer from the Licensor to exercise the + Licensed Rights under the terms and conditions of this + Public License. + + b. Additional offer from the Licensor -- Adapted Material. + Every recipient of Adapted Material from You + automatically receives an offer from the Licensor to + exercise the Licensed Rights in the Adapted Material + under the conditions of the Adapter's License You apply. + + c. No downstream restrictions. You may not offer or impose + any additional or different terms or conditions on, or + apply any Effective Technological Measures to, the + Licensed Material if doing so restricts exercise of the + Licensed Rights by any recipient of the Licensed + Material. + + 6. No endorsement. Nothing in this Public License constitutes or + may be construed as permission to assert or imply that You + are, or that Your use of the Licensed Material is, connected + with, or sponsored, endorsed, or granted official status by, + the Licensor or others designated to receive attribution as + provided in Section 3(a)(1)(A)(i). + + b. Other rights. + + 1. Moral rights, such as the right of integrity, are not + licensed under this Public License, nor are publicity, + privacy, and/or other similar personality rights; however, to + the extent possible, the Licensor waives and/or agrees not to + assert any such rights held by the Licensor to the limited + extent necessary to allow You to exercise the Licensed + Rights, but not otherwise. + + 2. Patent and trademark rights are not licensed under this + Public License. + + 3. To the extent possible, the Licensor waives any right to + collect royalties from You for the exercise of the Licensed + Rights, whether directly or through a collecting society + under any voluntary or waivable statutory or compulsory + licensing scheme. In all other cases the Licensor expressly + reserves any right to collect such royalties. + + +Section 3 -- License Conditions. + +Your exercise of the Licensed Rights is expressly made subject to the +following conditions. + + a. Attribution. + + 1. If You Share the Licensed Material (including in modified + form), You must: + + a. retain the following if it is supplied by the Licensor + with the Licensed Material: + + i. identification of the creator(s) of the Licensed + Material and any others designated to receive + attribution, in any reasonable manner requested by + the Licensor (including by pseudonym if + designated); + + ii. a copyright notice; + + iii. a notice that refers to this Public License; + + iv. a notice that refers to the disclaimer of + warranties; + + v. a URI or hyperlink to the Licensed Material to the + extent reasonably practicable; + + b. indicate if You modified the Licensed Material and + retain an indication of any previous modifications; and + + c. indicate the Licensed Material is licensed under this + Public License, and include the text of, or the URI or + hyperlink to, this Public License. + + 2. You may satisfy the conditions in Section 3(a)(1) in any + reasonable manner based on the medium, means, and context in + which You Share the Licensed Material. For example, it may be + reasonable to satisfy the conditions by providing a URI or + hyperlink to a resource that includes the required + information. + + 3. If requested by the Licensor, You must remove any of the + information required by Section 3(a)(1)(A) to the extent + reasonably practicable. + + b. ShareAlike. + + In addition to the conditions in Section 3(a), if You Share + Adapted Material You produce, the following conditions also apply. + + 1. The Adapter's License You apply must be a Creative Commons + license with the same License Elements, this version or + later, or a BY-SA Compatible License. + + 2. You must include the text of, or the URI or hyperlink to, the + Adapter's License You apply. You may satisfy this condition + in any reasonable manner based on the medium, means, and + context in which You Share Adapted Material. + + 3. You may not offer or impose any additional or different terms + or conditions on, or apply any Effective Technological + Measures to, Adapted Material that restrict exercise of the + rights granted under the Adapter's License You apply. + + +Section 4 -- Sui Generis Database Rights. + +Where the Licensed Rights include Sui Generis Database Rights that +apply to Your use of the Licensed Material: + + a. for the avoidance of doubt, Section 2(a)(1) grants You the right + to extract, reuse, reproduce, and Share all or a substantial + portion of the contents of the database; + + b. if You include all or a substantial portion of the database + contents in a database in which You have Sui Generis Database + Rights, then the database in which You have Sui Generis Database + Rights (but not its individual contents) is Adapted Material, + + including for purposes of Section 3(b); and + c. You must comply with the conditions in Section 3(a) if You Share + all or a substantial portion of the contents of the database. + +For the avoidance of doubt, this Section 4 supplements and does not +replace Your obligations under this Public License where the Licensed +Rights include other Copyright and Similar Rights. + + +Section 5 -- Disclaimer of Warranties and Limitation of Liability. + + a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE + EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS + AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF + ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, + IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, + WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR + PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, + ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT + KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT + ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. + + b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE + TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, + NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, + INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, + COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR + USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN + ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR + DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR + IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. + + c. The disclaimer of warranties and limitation of liability provided + above shall be interpreted in a manner that, to the extent + possible, most closely approximates an absolute disclaimer and + waiver of all liability. + + +Section 6 -- Term and Termination. + + a. This Public License applies for the term of the Copyright and + Similar Rights licensed here. However, if You fail to comply with + this Public License, then Your rights under this Public License + terminate automatically. + + b. Where Your right to use the Licensed Material has terminated under + Section 6(a), it reinstates: + + 1. automatically as of the date the violation is cured, provided + it is cured within 30 days of Your discovery of the + violation; or + + 2. upon express reinstatement by the Licensor. + + For the avoidance of doubt, this Section 6(b) does not affect any + right the Licensor may have to seek remedies for Your violations + of this Public License. + + c. For the avoidance of doubt, the Licensor may also offer the + Licensed Material under separate terms or conditions or stop + distributing the Licensed Material at any time; however, doing so + will not terminate this Public License. + + d. Sections 1, 5, 6, 7, and 8 survive termination of this Public + License. + + +Section 7 -- Other Terms and Conditions. + + a. The Licensor shall not be bound by any additional or different + terms or conditions communicated by You unless expressly agreed. + + b. Any arrangements, understandings, or agreements regarding the + Licensed Material not stated herein are separate from and + independent of the terms and conditions of this Public License. + + +Section 8 -- Interpretation. + + a. For the avoidance of doubt, this Public License does not, and + shall not be interpreted to, reduce, limit, restrict, or impose + conditions on any use of the Licensed Material that could lawfully + be made without permission under this Public License. + + b. To the extent possible, if any provision of this Public License is + deemed unenforceable, it shall be automatically reformed to the + minimum extent necessary to make it enforceable. If the provision + cannot be reformed, it shall be severed from this Public License + without affecting the enforceability of the remaining terms and + conditions. + + c. No term or condition of this Public License will be waived and no + failure to comply consented to unless expressly agreed to by the + Licensor. + + d. Nothing in this Public License constitutes or may be interpreted + as a limitation upon, or waiver of, any privileges and immunities + that apply to the Licensor or You, including from the legal + processes of any jurisdiction or authority. + + +======================================================================= + +Creative Commons is not a party to its public +licenses. Notwithstanding, Creative Commons may elect to apply one of +its public licenses to material it publishes and in those instances +will be considered the “Licensor.” The text of the Creative Commons +public licenses is dedicated to the public domain under the CC0 Public +Domain Dedication. Except for the limited purpose of indicating that +material is shared under a Creative Commons public license or as +otherwise permitted by the Creative Commons policies published at +creativecommons.org/policies, Creative Commons does not authorize the +use of the trademark "Creative Commons" or any other trademark or logo +of Creative Commons without its prior written consent including, +without limitation, in connection with any unauthorized modifications +to any of its public licenses or any other arrangements, +understandings, or agreements concerning use of licensed material. For +the avoidance of doubt, this paragraph does not form part of the +public licenses. + +Creative Commons may be contacted at creativecommons.org. diff --git a/config.yml b/_config.yml similarity index 76% rename from config.yml rename to _config.yml index 3d59ecd..9e1bfd3 100644 --- a/config.yml +++ b/_config.yml @@ -2,5 +2,6 @@ title: Go database/sql tutorial description: Use database/sql effectively! safe: true lsi: false -pygments: true +highlighter: rouge timezone: UTC +markdown: kramdown diff --git a/_data/nav.yml b/_data/nav.yml new file mode 100644 index 0000000..dfab5a9 --- /dev/null +++ b/_data/nav.yml @@ -0,0 +1,40 @@ +docs: + + - title: Go database/sql tutorial + url: index.html + + - title: Overview + url: overview.html + + - title: Importing a Database Driver + url: importing.html + + - title: Accessing the Database + url: accessing.html + + - title: Retrieving Result Sets + url: retrieving.html + + - title: Modifying Data and Using Transactions + url: modifying.html + + - title: Using Prepared Statements + url: prepared.html + + - title: Handling Errors + url: errors.html + + - title: Working with NULLs + url: nulls.html + + - title: Working with Unknown Columns + url: varcols.html + + - title: The Connection Pool + url: connection-pool.html + + - title: Surprises, Antipatterns and Limitations + url: surprises.html + + - title: Related Reading and Resources + url: references.html diff --git a/_includes/leftnav.html b/_includes/leftnav.html index 5ca5d57..832c653 100644 --- a/_includes/leftnav.html +++ b/_includes/leftnav.html @@ -1,22 +1,14 @@
diff --git a/_layouts/article.html b/_layouts/article.html index c2940a1..e80f700 100644 --- a/_layouts/article.html +++ b/_layouts/article.html @@ -7,34 +7,40 @@ url: index.html articles: - - title: Overview of Go's database/sql Package + title: Overview url: overview.html - - title: Importing a Database Driver + title: Importing a Driver url: importing.html - - title: Accessing the Database + title: Accessing a DB url: accessing.html - - title: Retrieving Result Sets + title: Getting Results url: retrieving.html - - title: Modifying Data and Using Transactions + title: Updates and Transactions url: modifying.html + - + title: Prepared Statements + url: prepared.html + - + title: Handling Errors + url: errors.html - title: Working with NULLs url: nulls.html - - title: Working with Unknown Columns + title: Unknown Columns url: varcols.html - - title: The Connection Pool + title: Connection Pooling url: connection-pool.html - - title: Surprises, Antipatterns and Limitations + title: Surprises and Limitations url: surprises.html - - title: Related Reading and Resources + title: Resources url: references.html --- @@ -48,11 +54,11 @@
+for rows.Next() {
+ // ...
+}
+if err = rows.Err(); err != nil {
+ // handle the error here
+}
+
+
+The error from `rows.Err()` could be the result of a variety of errors in the
+`rows.Next()` loop. The loop
+might exit for some reason other than finishing the loop normally, so you always
+need to check whether the loop terminated normally or not. An abnormal
+termination automatically calls `rows.Close()`, although it's harmless to call it
+multiple times.
+
+Errors From Closing Resultsets
+==============================
+
+You should always explicitly close a `sql.Rows` if you exit the loop
+prematurely, as previously mentioned. It's auto-closed if the loop exits
+normally or through an error, but you might mistakenly do this:
+
+
+for rows.Next() {
+ // ...
+ break; // whoops, rows is not closed! memory leak...
+}
+// do the usual "if err = rows.Err()" [omitted here]...
+// it's always safe to [re?]close here:
+if err = rows.Close(); err != nil {
+ // but what should we do if there's an error?
+ log.Println(err)
+}
+
+
+The error returned by `rows.Close()` is the only exception to the general rule
+that it's best to capture and check for errors in all database operations. If
+`rows.Close()` returns an error, it's unclear what you should do.
+Logging the error message or panicing might be the only sensible thing,
+and if that's not sensible, then perhaps you should just ignore the error.
+
+Errors From QueryRow()
+======================
+
+Consider the following code to fetch a single row:
+
+
+var name string
+err = db.QueryRow("select name from users where id = ?", 1).Scan(&name)
+if err != nil {
+ log.Fatal(err)
+}
+fmt.Println(name)
+
+
+What if there was no user with `id = 1`? Then there would be no row in the
+result, and `.Scan()` would not scan a value into `name`. What happens then?
+
+Go defines a special error constant, called `sql.ErrNoRows`, which is returned
+from `QueryRow()` when the result is empty. This needs to be handled as a
+special case in most circumstances. An empty result is often not considered an
+error by application code, and if you don't check whether an error is this
+special constant, you'll cause application-code errors you didn't expect.
+
+Errors from the query are deferred until `Scan()` is called, and then are
+returned from that. The above code is better written like this instead:
+
+
+var name string
+err = db.QueryRow("select name from users where id = ?", 1).Scan(&name)
+if err != nil {
+ if err == sql.ErrNoRows {
+ // there were no rows, but otherwise no error occurred
+ } else {
+ log.Fatal(err)
+ }
+}
+fmt.Println(name)
+
+
+One might ask why an empty result set is considered an error. There's nothing
+erroneous about an empty set. The reason is that the `QueryRow()` method needs
+to use this special-case in order to let the caller distinguish whether
+`QueryRow()` in fact found a row; without it, `Scan()` wouldn't do anything and
+you might not realize that your variable didn't get any value from the database
+after all.
+
+You should only run into this error when you're using `QueryRow()`. If you
+encounter this error elsewhere, you're doing something wrong.
+
+Identifying Specific Database Errors
+====================================
+
+It can be tempting to write code like the following:
+
+
+rows, err := db.Query("SELECT someval FROM sometable")
+// err contains:
+// ERROR 1045 (28000): Access denied for user 'foo'@'::1' (using password: NO)
+if strings.Contains(err.Error(), "Access denied") {
+ // Handle the permission-denied error
+}
+
+
+This is not the best way to do it, though. For example, the string value might
+vary depending on what language the server uses to send error messages. It's
+much better to compare error numbers to identify what a specific error is.
+
+The mechanism to do this varies between drivers, however, because this isn't
+part of `database/sql` itself. In the MySQL driver that this tutorial focuses
+on, you could write the following code:
+
+
+if driverErr, ok := err.(*mysql.MySQLError); ok { // Now the error number is accessible directly
+ if driverErr.Number == 1045 {
+ // Handle the permission-denied error
+ }
+}
+
+
+Again, the `MySQLError` type here is provided by this specific driver, and the
+`.Number` field may differ between drivers. The value of the number, however,
+is taken from MySQL's error message, and is therefore database specific, not
+driver specific.
+
+This code is still ugly. Comparing to 1045, a magic number, is a code smell.
+Some drivers (though not the MySQL one, for reasons that are off-topic here)
+provide a list of error identifiers. The Postgres `pq` driver does, for example, in
+[error.go](https://github.com/lib/pq/blob/master/error.go). And there's an
+external package of [MySQL error numbers maintained by
+VividCortex](https://github.com/VividCortex/mysqlerr). Using such a list, the
+above code is better written thus:
+
+
+if driverErr, ok := err.(*mysql.MySQLError); ok {
+ if driverErr.Number == mysqlerr.ER_ACCESS_DENIED_ERROR {
+ // Handle the permission-denied error
+ }
+}
+
+
+Handling Connection Errors
+==========================
+
+What if your connection to the database is dropped, killed, or has an error?
+
+You don't need to implement any logic to retry failed statements when this
+happens. As part of the [connection pooling](connection-pool.html) in
+`database/sql`, handling failed connections is built-in. If you execute a query
+or other statement and the underlying connection has a failure, Go will reopen a
+new connection (or just get another from the connection pool) and retry, up to
+10 times.
+
+There can be some unintended consequences, however. Some types of errors may be
+retried when other error conditions happen. This might also be driver-specific.
+One example that has occurred with the MySQL driver is that using `KILL` to
+cancel an undesired statement (such as a long-running query) results in the
+statement being retried up to 10 times.
+
+**Previous: [Using Prepared Statements](prepared.html)**
+**Next: [Working with NULLs](nulls.html)**
diff --git a/importing.md b/importing.md
index 9b6608a..ef1ab3a 100644
--- a/importing.md
+++ b/importing.md
@@ -30,7 +30,7 @@ import (
Notice that we're loading the driver anonymously, aliasing its package qualifier
to `_` so none of its exported names are visible to our code. Under the hood,
the driver registers itself as being available to the `database/sql` package,
-but in general nothing else happens.
+but in general nothing else happens with the exception that the init function is run.
Now you're ready to access a database.
diff --git a/modifying.md b/modifying.md
index 324a961..487aac3 100644
--- a/modifying.md
+++ b/modifying.md
@@ -12,7 +12,7 @@ Statements that Modify Data
===========================
Use `Exec()`, preferably with a prepared statement, to accomplish an `INSERT`,
-`UPDATE`, `DELETE`, or other statement that doesn't return rows. The following
+`UPDATE`, `DELETE`, or another statement that doesn't return rows. The following
example shows how to insert a row and inspect metadata about the operation:
@@ -73,8 +73,7 @@ with that transaction. The methods on the `Tx` map one-for-one to methods you can call on the database itself, such as `Query()` and so forth. Prepared statements that are created in a transaction are bound exclusively to -that transaction, and can't be used outside of it. Likewise, prepared statements -created only on a database handle can't be used within a transaction. +that transaction. See [prepared statements](prepared.html) for more. You should not mingle the use of transaction-related functions such as `Begin()` and `Commit()` with SQL statements such as `BEGIN` and `COMMIT` in your SQL @@ -84,5 +83,21 @@ code. Bad things might result: * The state of the database could get out of sync with the state of the Go variables representing it. * You could believe you're executing queries on a single connection, inside of a transaction, when in reality Go has created several connections for you invisibly and some statements aren't part of the transaction. +While you are working inside a transaction you should be careful not to make +calls to the `db` variable. Make all of your calls to the `Tx` variable that you +created with `db.Begin()`. `db` is not in a transaction, only the `Tx` object is. +If you make further calls to `db.Exec()` or similar, those will happen outside +the scope of your transaction, on other connections. + +If you need to work with multiple statements that modify connection state, you +need a `Tx` even if you don't want a transaction per se. For example: + +* Creating temporary tables, which are only visible to one connection. +* Setting variables, such as MySQL's `SET @var := somevalue` syntax. +* Changing connection options, such as character sets or timeouts. + +If you need to do any of these things, you need to bind your activity to a +single connection, and the only way to do that in Go is to use a `Tx`. + **Previous: [Retrieving Result Sets](retrieving.html)** -**Next: [Working with NULLs](nulls.html)** +**Next: [Using Prepared Statements](prepared.html)** diff --git a/nulls.md b/nulls.md index fc18831..389315c 100644 --- a/nulls.md +++ b/nulls.md @@ -37,5 +37,22 @@ you need more convincing: If you need to define your own types to handle NULLs, you can copy the design of `sql.NullString` to achieve that. -**Previous: [Modifying Data and Using Transactions](modifying.html)** +If you can't avoid having NULL values in your database, there is another work around that most database systems support, namely `COALESCE()`. Something like the following might be something that you can use, without introducing a myriad of `sql.Null*` types. +Under the hood, `db.Query()` actually prepares, executes, and closes a prepared statement. That's three round-trips to the database. If you're not careful, you can triple the number of database interactions your application makes! Some -drivers can avoid this in specific cases with an addition to `database/sql` in -Go 1.1, but not all drivers are smart enough to do that. Caveat Emptor. - -Naturally prepared statements and the management of prepared statements cost -resources. You should take care to close statements when they are not used again. +drivers can avoid this in specific cases, +but not all drivers do. See [prepared statements](prepared.html) for more. Single-Row Queries ================== @@ -146,6 +167,7 @@ stmt, err := db.Prepare("select name from users where id = ?") if err != nil { log.Fatal(err) } +defer stmt.Close() var name string err = stmt.QueryRow(1).Scan(&name) if err != nil { @@ -154,21 +176,5 @@ if err != nil { fmt.Println(name) -Go defines a special error constant, called `sql.ErrNoRows`, which is returned -from `QueryRow()` when the result is empty. This needs to be handled as a -special case in most circumstances. An empty result is often not considered an -error by application code, and if you don't check whether an error is this -special constant, you'll cause application-code errors you didn't expect. - -One might ask why an empty result set is considered an error. There's nothing -erroneous about an empty set. The reason is that the `QueryRow()` method needs -to use this special-case in order to let the caller distinguish whether -`QueryRow()` in fact found a row; without it, `Scan()` wouldn't do anything and -you might not realize that your variable didn't get any value from the database -after all. - -You should not run into this error when you're not using `QueryRow()`. If you -encounter this error elsewhere, you're doing something wrong. - **Previous: [Accessing the Database](accessing.html)** **Next: [Modifying Data and Using Transactions](modifying.html)** diff --git a/surprises.md b/surprises.md index f20cc2d..359cbe4 100644 --- a/surprises.md +++ b/surprises.md @@ -17,7 +17,7 @@ resources or preventing them from being reused effectively: * Opening and closing databases can cause exhaustion of resources. * Failing to read all rows or use `rows.Close()` reserves connections from the pool. * Using `Query()` for a statement that doesn't return rows will reserve a connection from the pool. -* Failing to use prepared statements can lead to a lot of extra database activity. +* Failing to be aware of how [prepared statements](prepared.html) work can lead to a lot of extra database activity. Large uint64 Values =================== @@ -26,7 +26,7 @@ Here's a surprising error. You can't pass big unsigned integers as parameters to statements if their high bit is set:+rows, err := db.Query(` + SELECT + name, + COALESCE(other_field, '') as otherField + WHERE id = ? +`, 42) + +for rows.Next() { + err := rows.Scan(&name, &otherField) + // .. + // If `other_field` was NULL, `otherField` is now an empty string. This works with other data types as well. +} ++ + +**Previous: [Handling Errors](errors.html)** **Next: [Working with Unknown Columns](varcols.html)** diff --git a/overview.md b/overview.md index 92d344a..337973b 100644 --- a/overview.md +++ b/overview.md @@ -12,7 +12,7 @@ of a "database" or "schema." It's an abstraction of the interface and existence of a database, which might be as varied as a local file, accessed through a network connection, or in-memory and in-process. -The `sql.DB` performs some important tasks for you behind the scenes: +`sql.DB` performs some important tasks for you behind the scenes: * It opens and closes connections to the actual underlying database, via the driver. * It manages a pool of connections as needed, which may be a variety of things as mentioned. @@ -21,7 +21,7 @@ The `sql.DB` abstraction is designed to keep you from worrying about how to manage concurrent access to the underlying datastore. A connection is marked in-use when you use it to perform a task, and then returned to the available pool when it's not in use anymore. One consequence of this is that **if you fail -to release connections back to the pool, you can cause `db.SQL` to open a lot of +to release connections back to the pool, you can cause `sql.DB` to open a lot of connections**, potentially running out of resources (too many connections, too many open file handles, lack of available network ports, etc). We'll discuss more about this later. diff --git a/prepared.md b/prepared.md new file mode 100644 index 0000000..e382940 --- /dev/null +++ b/prepared.md @@ -0,0 +1,133 @@ +--- +layout: article +title: Using Prepared Statements +--- + +Prepared statements have all the usual benefits in Go: security, efficiency, +convenience. But the way they're implemented is a little different from what +you might be used to, especially with regards to how they interact with some of +the internals of `database/sql`. + +Prepared Statements And Connections +=================================== + +At the database level, a prepared statement is bound to a single database +connection. The typical flow is that the client sends a SQL statement with +placeholders to the server for preparation, the server responds with a statement +ID, and then the client executes the statement by sending its ID and parameters. + +In Go, however, connections are not exposed directly to the user of the +`database/sql` package. You don't prepare a statement on a connection. You +prepare it on a `DB` or a `Tx`. And `database/sql` has some convenience +behaviors such as automatic retries. For these reasons, the underlying +association between prepared statements and connections, which exists at the +driver level, is hidden from your code. + +Here's how it works: + +1. When you prepare a statement, it's prepared on a connection in the pool. +2. The `Stmt` object remembers which connection was used. +3. When you execute the `Stmt`, it tries to use the connection. If it's not + available because it's closed or busy doing something else, it gets another + connection from the pool *and re-prepares the statement with the database on + another connection.* + +Because statements will be re-prepared as needed when their original connection +is busy, it's possible for high-concurrency usage of the database, which may +keep a lot of connections busy, to create a large number of prepared statements. +This can result in apparent leaks of statements, statements being prepared and +re-prepared more often than you think, and even running into server-side limits +on the number of statements. + +Avoiding Prepared Statements +============================ + +Go creates prepared statements for you under the covers. A simple +`db.Query(sql, param1, param2)`, for example, works by preparing the sql, then +executing it with the parameters and finally closing the statement. + +Sometimes a prepared statement is not what you want, however. There might be +several reasons for this: + +1. The database doesn't support prepared statements. When using the MySQL + driver, for example, you can connect to MemSQL and Sphinx, because they + support the MySQL wire protocol. But they don't support the "binary" protocol + that includes prepared statements, so they can fail in confusing ways. +2. The statements aren't reused enough to make them worthwhile, and security + issues are handled in other ways, so performance overhead is undesired. An + example of this can be seen at the + [VividCortex blog](https://vividcortex.com/blog/2014/11/19/analyzing-prepared-statement-performance-with-vividcortex/). + +If you don't want to use a prepared statement, you need to use `fmt.Sprint()` or +similar to assemble the SQL, and pass this as the only argument to `db.Query()` +or `db.QueryRow()`. And your driver needs to support plaintext query execution, +which is added in Go 1.1 via the `Execer` and `Queryer` interfaces, +[documented here](http://golang.org/pkg/database/sql/driver/#Execer). + +Prepared Statements in Transactions +=================================== + +Prepared statements that are created in a `Tx` are bound exclusively to +it, so the earlier cautions about repreparing do not apply. When +you operate on a `Tx` object, your actions map directly to the one and only one +connection underlying it. + +This also means that prepared statements created inside a `Tx` can't be used +separately from it. Likewise, prepared statements created on a `DB` can't be +used within a transaction, because they will be bound to a different connection. + +To use a prepared statement prepared outside the transaction in a `Tx`, you can use +`Tx.Stmt()`, which will create a new transaction-specific statement from the one +prepared outside the transaction. It does this by taking an existing prepared statement, +setting the connection to that of the transaction and repreparing all statements every +time they are executed. This behavior and its implementation are undesirable and there's +even a TODO in the `database/sql` source code to improve it; we advise against using this. + +Caution must be exercised when working with prepared statements in +transactions. Consider the following example: + ++tx, err := db.Begin() +if err != nil { + log.Fatal(err) +} +defer tx.Rollback() +stmt, err := tx.Prepare("INSERT INTO foo VALUES (?)") +if err != nil { + log.Fatal(err) +} +defer stmt.Close() // danger! +for i := 0; i < 10; i++ { + _, err = stmt.Exec(i) + if err != nil { + log.Fatal(err) + } +} +err = tx.Commit() +if err != nil { + log.Fatal(err) +} +// stmt.Close() runs here! ++ + +Before Go 1.4 closing a `*sql.Tx` released the connection associated with it back into the +pool, but the deferred call to Close on the prepared statement was executed +**after** that has happened, which could lead to concurrent access to the +underlying connection, rendering the connection state inconsistent. +If you use Go 1.4 or older, you should make sure the statement is always closed before the transaction is +committed or rolled back. [This issue](https://github.com/golang/go/issues/4459) was fixed in Go 1.4 by [CR 131650043](https://codereview.appspot.com/131650043). + +Parameter Placeholder Syntax +============================ + +The syntax for placeholder parameters in prepared statements is +database-specific. For example, comparing MySQL, PostgreSQL, and Oracle: + + MySQL PostgreSQL Oracle + ===== ========== ====== + WHERE col = ? WHERE col = $1 WHERE col = :col + VALUES(?, ?, ?) VALUES($1, $2, $3) VALUES(:val1, :val2, :val3) + +**Previous: [Modifying Data and Using Transactions](modifying.html)** +**Next: [Handling Errors](errors.html)** diff --git a/references.md b/references.md index 86ca6da..b61fcf6 100644 --- a/references.md +++ b/references.md @@ -8,6 +8,8 @@ Here are some external sources of information we've found to be helpful. * [http://golang.org/pkg/database/sql/](http://golang.org/pkg/database/sql/) * [http://jmoiron.net/blog/gos-database-sql/](http://jmoiron.net/blog/gos-database-sql/) * [http://jmoiron.net/blog/built-in-interfaces/](http://jmoiron.net/blog/built-in-interfaces/) +* The VividCortex blog, e.g. [transparent encryption](https://vividcortex.com/blog/2014/11/11/encrypting-data-in-mysql-with-go/) +* Overview of SQL Drivers from [golang github](https://github.com/golang/go/wiki/SQLDrivers) We hope you've found this website to be helpful. If you have any suggestions for improvements, please send pull requests or open an issue report at diff --git a/retrieving.md b/retrieving.md index d774a7e..33cee55 100644 --- a/retrieving.md +++ b/retrieving.md @@ -54,36 +54,57 @@ Here's what's happening in the above code: 4. We read the columns in each row into variables with `rows.Scan()`. 5. We check for errors after we're done iterating over the rows. -A couple parts of this are easy to get wrong, and can have bad consequences. - -First, as long as there's an open result set (represented by `rows`), the -underlying connection is busy and can't be used for any other query. That means -it's not available in the connection pool. If you iterate over all of the rows -with `rows.Next()`, eventually you'll read the last row, and `rows.Next()` will -encounter an internal EOF error and call `rows.Close()` for you. But if for any -reason you exit that loop -- an error, an early return, or so on -- then the -`rows` doesn't get closed, and the connection remains open. This is an easy way -to run out of resources. This is why **you should always `defer rows.Close()`**, -even if you also call it explicitly at the end of the loop, which isn't a bad -idea. `rows.Close()` is a harmless no-op if it's already closed, so you can call -it multiple times. Notice, however, that we check the error first, and only do -`rows.Close()` if there isn't an error, in order to avoid a runtime panic. - -Second, you should always check for an error at the end of the `for rows.Next()` -loop. If there's an error during the loop, you need to know about it. Don't just -assume that the loop iterates until you've processed all the rows. - -The error returned by `rows.Close()` is the only exception to the general rule -that it's best to capture and check for errors in all database operations. If -`rows.Close()` throws an error, it's unclear what is the right thing to do. -Logging the error message or panicing might be the only sensible thing to do, -and if that's not sensible, then perhaps you should just ignore the error. - This is pretty much the only way to do it in Go. You can't get a row as a map, for example. That's because everything is strongly typed. You need to create variables of the correct type and pass pointers to them, as shown. +A couple parts of this are easy to get wrong, and can have bad consequences. + +* You should always check for an error at the end of the `for rows.Next()` + loop. If there's an error during the loop, you need to know about it. Don't + just assume that the loop iterates until you've processed all the rows. +* Second, as long as there's an open result set (represented by `rows`), the + underlying connection is busy and can't be used for any other query. That + means it's not available in the connection pool. If you iterate over all of + the rows with `rows.Next()`, eventually you'll read the last row, and + `rows.Next()` will encounter an internal EOF error and call `rows.Close()` for + you. But if for some reason you exit that loop -- an early return, or so on -- + then the `rows` doesn't get closed, and the connection remains open. (It is + auto-closed if `rows.Next()` returns false due to an error, though). This is + an easy way to run out of resources. +* `rows.Close()` is a harmless no-op if it's already closed, so you can call + it multiple times. Notice, however, that we check the error first, and only + call `rows.Close()` if there isn't an error, in order to avoid a runtime panic. +* You should **always `defer rows.Close()`**, even if you also call `rows.Close()` + explicitly at the end of the loop, which isn't a bad idea. +* Don't `defer` within a loop. A deferred statement doesn't get executed until + the function exits, so a long-running function shouldn't use it. If you do, + you will slowly accumulate memory. If you are repeatedly querying and + consuming result sets within a loop, you should explicitly call `rows.Close()` + when you're done with each result, and not use `defer`. + +How Scan() Works +================ + +When you iterate over rows and scan them into destination variables, Go performs data +type conversions work for you, behind the scenes. It is based on the type of the +destination variable. Being aware of this can clean up your code and help avoid +repetitive work. + +For example, suppose you select some rows from a table that is defined with +string columns, such as `VARCHAR(45)` or similar. You happen to know, however, +that the table always contains numbers. If you pass a pointer to a string, Go +will copy the bytes into the string. Now you can use `strconv.ParseInt()` or +similar to convert the value to a number. You'll have to check for errors in the +SQL operations, as well as errors parsing the integer. This is messy and +tedious. + +Or, you can just pass `Scan()` a pointer to an integer. Go will detect that and +call `strconv.ParseInt()` for you. If there's an error in conversion, the call +to `Scan()` will return it. Your code is neater and smaller now. This is the +recommended way to use `database/sql`. + Preparing Queries ================= @@ -112,16 +133,16 @@ defer rows.Close() for rows.Next() { // ... } +if err = rows.Err(); err != nil { + log.Fatal(err) +}
-_, err := db.Exec("INSERT INTO users(id) VALUES", math.MaxUint64)
+_, err := db.Exec("INSERT INTO users(id) VALUES", math.MaxUint64) // Error
This will throw an error. Be careful if you use `uint64` values, as they may
@@ -59,21 +59,15 @@ Database-Specific Syntax
========================
The `database/sql` API provides an abstraction of a row-oriented database, but
-specific databases and drivers can differ in behavior and/or syntax. One
-example is the syntax for placeholder parameters in prepared statements. For
-example, comparing MySQL, PostgreSQL, and Oracle:
-
- MySQL PostgreSQL Oracle
- ===== ========== ======
- WHERE col = ? WHERE col = $1 WHERE col = :col
- VALUES(?, ?, ?) VALUES($1, $2, $3) VALUES(:val1, :val2, :val3)
+specific databases and drivers can differ in behavior and/or syntax, such as
+[prepared statement placeholders](prepared.html).
Multiple Result Sets
====================
The Go driver doesn't support multiple result sets from a single query in any
way, and there doesn't seem to be any plan to do that, although there is [a
-feature request](https://code.google.com/p/go/issues/detail?id=5171) for
+feature request](https://github.com/golang/go/issues/5171) for
supporting bulk operations such as bulk copy.
This means, among other things, that a stored procedure that returns multiple
@@ -87,7 +81,7 @@ be done at present. It might seem that you'd be able to call a simple
procedure that returns a single result set, by executing something like this:
-err := db.QueryRow("CALL mydb.myprocedure").Scan(&result)
+err := db.QueryRow("CALL mydb.myprocedure").Scan(&result) // Error
In fact, this won't work. You'll get the following error: _Error
@@ -104,7 +98,7 @@ The `database/sql` doesn't explicitly have multiple statement support, which mea
that the behavior of this is backend dependent:
-_, err := db.Exec("DELETE FROM tbl1; DELETE FROM tbl2")
+_, err := db.Exec("DELETE FROM tbl1; DELETE FROM tbl2") // Error/unpredictable result
The server is allowed to interpret this however it wants, which can include
@@ -113,11 +107,38 @@ returning an error, executing only the first statement, or executing both.
Similarly, there is no way to batch statements in a transaction. Each statement
in a transaction must be executed serially, and the resources in the results,
such as a Row or Rows, must be scanned or closed so the underlying connection is free
-for the next statement to use. Since transactions are connection state and bind
-to a single database connection, you may wind up with a corrupted connection if
-you attempt to perform another statement before the first has released that resource
-and cleaned up after itself. This also means that each statement in a transaction
-results in a separate set of network round-trips to the database.
+for the next statement to use. This differs from the usual behavior when you're
+not working with a transaction. In that scenario, it is perfectly possible to
+execute a query, loop over the rows, and within the loop make a query to the
+database (which will happen on a new connection):
+
+
+rows, err := db.Query("select * from tbl1") // Uses connection 1
+for rows.Next() {
+ err = rows.Scan(&myvariable)
+ // The following line will NOT use connection 1, which is already in-use
+ db.Query("select * from tbl2 where id = ?", myvariable)
+}
+
+
+But transactions are bound to
+just one connection, so this isn't possible with a transaction:
+
+
+tx, err := db.Begin()
+rows, err := tx.Query("select * from tbl1") // Uses tx's connection
+for rows.Next() {
+ err = rows.Scan(&myvariable)
+ // ERROR! tx's connection is already busy!
+ tx.Query("select * from tbl2 where id = ?", myvariable)
+}
+
+
+Go doesn't stop you from trying, though. For that reason, you may wind up with a
+corrupted connection if you attempt to perform another statement before the
+first has released its resources and cleaned up after itself. This also means
+that each statement in a transaction results in a separate set of network
+round-trips to the database.
**Previous: [The Connection Pool](connection-pool.html)**
**Next: [Related Reading and Resources](references.html)**