mirror of
https://github.com/neondatabase/neon.git
synced 2026-01-13 08:22:55 +00:00
## Problem For #11992 I realised we need to get the type info before executing the query. This is important to know how to decode rows with custom types, eg the following query: ```sql CREATE TYPE foo AS ENUM ('foo','bar','baz'); SELECT ARRAY['foo'::foo, 'bar'::foo, 'baz'::foo] AS data; ``` Getting that to work was harder that it seems. The original tokio-postgres setup has a split between `Client` and `Connection`, where messages are passed between. Because multiple clients were supported, each client message included a dedicated response channel. Each request would be terminated by the `ReadyForQuery` message. The flow I opted to use for parsing types early would not trigger a `ReadyForQuery`. The flow is as follows: ``` PARSE "" // parse the user provided query DESCRIBE "" // describe the query, returning param/result type oids FLUSH // force postgres to flush the responses early // wait for descriptions // check if we know the types, if we don't then // setup the typeinfo query and execute it against each OID: PARSE typeinfo // prepare our typeinfo query DESCRIBE typeinfo FLUSH // force postgres to flush the responses early // wait for typeinfo statement // for each OID we don't know: BIND typeinfo EXECUTE FLUSH // wait for type info, might reveal more OIDs to inspect // close the typeinfo query, we cache the OID->type map and this is kinder to pgbouncer. CLOSE typeinfo // finally once we know all the OIDs: BIND "" // bind the user provided query - already parsed - to the user provided params EXECUTE // run the user provided query SYNC // commit the transaction ``` ## Summary of changes Please review commit by commit. The main challenge was allowing one query to issue multiple sub-queries. To do this I first made sure that the client could fully own the connection, which required removing any shared client state. I then had to replace the way responses are sent to the client, by using only a single permanent channel. This required some additional effort to track which query is being processed. Lastly I had to modify the query/typeinfo functions to not issue `sync` commands, so it would fit into the desired flow above. To note: the flow above does force an extra roundtrip into each query. I don't know yet if this has a measurable latency overhead.
131 lines
4.0 KiB
Rust
131 lines
4.0 KiB
Rust
use std::pin::Pin;
|
|
use std::sync::Arc;
|
|
use std::task::{Context, Poll};
|
|
|
|
use fallible_iterator::FallibleIterator;
|
|
use futures_util::{Stream, ready};
|
|
use pin_project_lite::pin_project;
|
|
use postgres_protocol2::message::backend::Message;
|
|
use tracing::debug;
|
|
|
|
use crate::client::{InnerClient, Responses};
|
|
use crate::{Error, ReadyForQueryStatus, SimpleQueryMessage, SimpleQueryRow};
|
|
|
|
/// Information about a column of a single query row.
|
|
#[derive(Debug)]
|
|
pub struct SimpleColumn {
|
|
name: String,
|
|
}
|
|
|
|
impl SimpleColumn {
|
|
pub(crate) fn new(name: String) -> SimpleColumn {
|
|
SimpleColumn { name }
|
|
}
|
|
|
|
/// Returns the name of the column.
|
|
pub fn name(&self) -> &str {
|
|
&self.name
|
|
}
|
|
}
|
|
|
|
pub async fn simple_query<'a>(
|
|
client: &'a mut InnerClient,
|
|
query: &str,
|
|
) -> Result<SimpleQueryStream<'a>, Error> {
|
|
debug!("executing simple query: {}", query);
|
|
|
|
let responses = client.send_simple_query(query)?;
|
|
|
|
Ok(SimpleQueryStream {
|
|
responses,
|
|
columns: None,
|
|
status: ReadyForQueryStatus::Unknown,
|
|
})
|
|
}
|
|
|
|
pub async fn batch_execute(
|
|
client: &mut InnerClient,
|
|
query: &str,
|
|
) -> Result<ReadyForQueryStatus, Error> {
|
|
debug!("executing statement batch: {}", query);
|
|
|
|
let responses = client.send_simple_query(query)?;
|
|
|
|
loop {
|
|
match responses.next().await? {
|
|
Message::ReadyForQuery(status) => return Ok(status.into()),
|
|
Message::CommandComplete(_)
|
|
| Message::EmptyQueryResponse
|
|
| Message::RowDescription(_)
|
|
| Message::DataRow(_) => {}
|
|
_ => return Err(Error::unexpected_message()),
|
|
}
|
|
}
|
|
}
|
|
|
|
pin_project! {
|
|
/// A stream of simple query results.
|
|
pub struct SimpleQueryStream<'a> {
|
|
responses: &'a mut Responses,
|
|
columns: Option<Arc<[SimpleColumn]>>,
|
|
status: ReadyForQueryStatus,
|
|
}
|
|
}
|
|
|
|
impl SimpleQueryStream<'_> {
|
|
/// Returns if the connection is ready for querying, with the status of the connection.
|
|
///
|
|
/// This might be available only after the stream has been exhausted.
|
|
pub fn ready_status(&self) -> ReadyForQueryStatus {
|
|
self.status
|
|
}
|
|
}
|
|
|
|
impl Stream for SimpleQueryStream<'_> {
|
|
type Item = Result<SimpleQueryMessage, Error>;
|
|
|
|
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
|
|
let this = self.project();
|
|
loop {
|
|
match ready!(this.responses.poll_next(cx)?) {
|
|
Message::CommandComplete(body) => {
|
|
let rows = body
|
|
.tag()
|
|
.map_err(Error::parse)?
|
|
.rsplit(' ')
|
|
.next()
|
|
.unwrap()
|
|
.parse()
|
|
.unwrap_or(0);
|
|
return Poll::Ready(Some(Ok(SimpleQueryMessage::CommandComplete(rows))));
|
|
}
|
|
Message::EmptyQueryResponse => {
|
|
return Poll::Ready(Some(Ok(SimpleQueryMessage::CommandComplete(0))));
|
|
}
|
|
Message::RowDescription(body) => {
|
|
let columns = body
|
|
.fields()
|
|
.map(|f| Ok(SimpleColumn::new(f.name().to_string())))
|
|
.collect::<Vec<_>>()
|
|
.map_err(Error::parse)?
|
|
.into();
|
|
|
|
*this.columns = Some(columns);
|
|
}
|
|
Message::DataRow(body) => {
|
|
let row = match &this.columns {
|
|
Some(columns) => SimpleQueryRow::new(columns.clone(), body)?,
|
|
None => return Poll::Ready(Some(Err(Error::unexpected_message()))),
|
|
};
|
|
return Poll::Ready(Some(Ok(SimpleQueryMessage::Row(row))));
|
|
}
|
|
Message::ReadyForQuery(s) => {
|
|
*this.status = s.into();
|
|
return Poll::Ready(None);
|
|
}
|
|
_ => return Poll::Ready(Some(Err(Error::unexpected_message()))),
|
|
}
|
|
}
|
|
}
|
|
}
|