Devices willing to implement A2DP may offer several codecs, like SBC, AAC, aptX HD or LDAC. This excellent blog post goes into depth about the particularities of each one of these. The Bose headset only supports two codecs, namely SBC and AAC, so these are our options.

SIG created SBC (Low-complexity sub-band codec), a mandatory codec, so there is no risk of one device not talking with another because they don't implement the same set of codecs. SBC is very flexible and might provide a poor performance out of the box, but that wasn't my experience with Pipewire.

AAC (Advanced Audio Encoding) is a popular codec found in many videos and music on the web. In addition, Apple products, such as macOS, iOS, iTunes, and Apple Music, are famous for this codec support. It is less configurable than SBC, but it provides a better audio experience in theory.

For the Whatsapp calls, I chose AAC because it's the default after connecting to the headphone, but both would fit because my ordinary ears can't notice a difference between them.

HSP/HFP codec choices are stricter. The CVSD codec supports only audio at 8 kHz and the mSBC (wideband speech) at 16 kHz, with a single channel.

SIG only mandates devices to support the poorer CVSD, not mSBC. That's why the Bose headphone only supports CVSD, and the Linux computer needs to be the bridge between the smartphone and the headphone. If mSBC codec was supported, I could simply pick up the call via the headphone connected to the smartphone directly.

Using CVSD is a no-no because the audio is terrible for the caller and me, especially considering I talk with older relatives. Therefore, mSBC codec is the way to go.

Classic or BR/EDR (Basic Rate / Enhanced Data Rate) Bluetooth operates on the 2.4 GHz band and has adjacent channels to avoid signal interference, just like Wifi. Did you ever need to switch manually among the eleven channels on your router to run away from "crowded spaces"? In Bluetooth, this frequency hopping may happen hundreds of times every second among their 79 channels, and each of these hops switches the channels pseudo-randomly every 0.625ms (1600 times per second).

The clock of one of the devices called Central decides which channel to switch. All the devices following this hopping pattern are called Peripherals. The throughput can be 1Mbps (Basic Rate), 2Mbps or 3 Mbps (Enhanced Data Rate).

This video from Branch Education goes more in-depth about how the Bluetooth physical layer works. Actually, all the videos on this channel are superb and are worth a look.

The logical layer sits above the physical layer. It is responsible for managing the connections among devices, assigning which device is the central and the peripheral, and converting the raw bytes from the physical layer into frames. Ethernet has a similar structure with its physical and link layer. Three types of links can be established, ACL (Asynchronous Connection-Oriented), SCO (Synchronous Connection-Oriented) and eSCO (extended Synchronous Connection-Oriented).

SCO links reserves a certain amount of slots to guarantee a constant transmission rate. Besides having the same reserved slots, the newer eSCO links support a retransmission window to offer more reliability to the connection. Practically, bidirectional audio; a.k.a phone calls, uses SCO and eSCO.

The ACL links use the remaining slots not used by SCO/eSCO and leave the most complex part of multiplexing and ordering to a protocol in an above layer called L2CAP. But we'll get there eventually. ACL is used basically for everything else that's not a voice call, like listening to music, moving the mouse or even doing the handshake of the SCO/eSCO link.

In Linux, the controller layer code lives inside the hardware chip named Bluetooth controller, and it's generally a closed-source blob that lives inside the linux-firmware project. So when Intel wants to fix a bug or ship new functionality for my AX200, they update a targeted blob for the controller in this repo.

The host layer implements L2CAP (Logical Link Control and Adaptation Protocol) to make ACL more robust, which segments packets, adds error control and does not allow packets to overflow the ACL channel. It allows isochronous communication (in-order packets), necessary for a good audio experience. Other protocols, such as RFCOMM (used as a replacement for serial cables) and SDP (fundamental protocol useful for discovery among devices), sit on top of L2CAP.

In Linux, the kernel implements the host layer and the userspace communicates with it via sockets.

Be it with L2CAP and ACL or directly sending or receiving voice packets through SCO/eSCO, the host layer needs a way to communicate with the Bluetooth controller. To allow both pieces to talk to each other, SIG created the HCI (Host Controller Interface) protocol.

One of the ways that the Linux kernel implements the HCI layer is through the Linux USB API. The kernel encapsulates the incoming ACL/SCO packets into HCI packets and then to USB packets. The controller receives these USB packets and assembles them into ACL/SCO packets. As a sender, the controller performs the opposite flow. Even when using a Bluetooth keyboard or mouse in Linux, you're somehow using USB to make it work. How wild is that? (This might not be the case when the controller uses UART or RS232 for the HCI transport).

In Linux, the translation of ACL/SCO packets and USB packets happens in the btusb module.

The most relevant HCI packets for these calls are:

• Commands and Events: The host can modify the controller state or receive events. Similar to Netlink sockets changing the network configuration in Linux.
• Data packets: Send and receive ACL or SCO data

In the logs, I was seeing the line coming from a2dp-sink file: a2dp-sink 0x55ea222c72c8: Resource temporarily unavailable. This message is a readable error for EAGAIN with code error 11.

The send socket call with a EAGAIN error means that this non-blocking operation is refused, and the userspace counterpart should try again later. This behaviour also manifests itself on TCP/IP calls.

The kernel is rejecting the write in this part of code and the simplified version is shown below:

struct sk_buff *sock_alloc_send_pskb(struct sock *sk, int *errcode)
{
    if ((sk->sk_wmem_alloc - 1) < sk->sk_sndbuf)
        break;

    sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk);
    set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
    err = -EAGAIN;

    goto failure;

    skb = alloc_skb_with_frags(...);

    return skb;

failure:
    *errcode = err;
    return NULL;
}

A new sk_buff is allocated and connected to the socket only if sk_wmem_alloc field is smaller than sk_sndbuf field. When a new buffer arrives into the kernel, the sk_wmem_alloc increases its size.

By default, the socket sets the sk_sndbuf value from /proc/sys/net/core/wmem_default (212992 by default in my machine). But, Pipewire sets this socket property to a lower value with a setsockopt call passing the SO_SNDBUF parameter. PipeWire multiplies the write MTU of the device by two. As an example, the Bose headphone has an MTU of 875.

Naive me thought: "It's such a low value. I will increase the size of the buffer. That will solve it." So, instead of multiplying by two, I changed the PipeWire code to multiply by 5 to check what happened. However, it only made matters worse because when the EAGAIN error happened, the audio didn't catch up, and I could only hear silence after the first hiccup. Then some audio after some seconds and then silence again. I couldn't find the reasoning for setting a low buffer on the PipeWire codebase. Still, I could trace back why the two-factor multiplication is there from a Pulseaudio commit and the forum discussion.

Pulseaudio/PipeWire decreases the buffer size to avoid lags after "temporary connection drops". Logically, the error is one layer below, and I needed to check why the buffers were not emptied on time.

I hit a wall. Looking at the BlueZ kernel code, I didn't know why the socket was full and not accepting new buffers. And why is this happening only occasionally?! Looking at logs wasn't helping me much, and I needed a new approach.

That's when I stumbled upon eBPF (Extended Berkeley Packet Filter). eBPF is a recent technology that allows extending the kernel without recompilation or adding new modules. It supports many features, one of which is to plug some hooks into functions and log their parameters. This allowed me to get my feet wet starting with stackcount that checks the number of invocations of a function and a rudimentary stack trace. Then I started creating my own scripts to log some interesting functions.

Looking at logs from different functions simultaneously proved to be hard to follow. Besides, it was difficult to extract historical data to compare when everything was fine and days when nothing worked. One thing led to another, and I created bluetooth_exporter as a tool to help me solve this issue.

This tool exports Prometheus data from what's happening on the Bluetooth layer in the kernel. The repo also includes a docker-compose with Prometheus and Grafana setup for easy integration. Besides other metrics, I could see:

• The number of write syscalls from Bluetooth sockets and their return codes
• The time an HCI packet takes to pass through the USB layer
• The interval that an ACL packet takes to be acknowledged

The repo README provides a more in-depth explanation of these features and where they're hooked on the kernel. I'm pretty satisfied because it's generic enough to be used in other contexts, like correlating the codec change with the Bluetooth throughput. Keep in mind that nothing in this repo is PipeWire specific.

Before jumping to conclusions, we need to understand some of the layers an ACL packet needs to pass before eventually reaching the controller.

1. When a Bluetooth controller is being initialized, the kernel sends a HCI_Read_Buffer_Size HCI command to the controller. The returned value signals the total packets that the controller can process concurrently. The kernel stores the ACL field in a field called acl_cnt. In the case of my controller (AX200), the value is 4 slots for ACL and 6 slots for SCO.
2. Whenever an ACL packet is enqueued, the kernel decreases the acl_cnt by 1. This represents that the controller is "busy" with that outgoing packet. In addition, when the ACL packet is scheduled, the sk_buff is destroyed and, consequently, the higher level sk_wmem_alloc decreased to accept a new buffer from userspace.
3. If a new packet arrives and the value of acl_cnt is 0, no new packet is sent to the controller.
4. Whenever an ACL packet is processed by the sink device (Bose headset), the controller sends a HCI_Number_Of_Completed_Packets event. Then, the acl_cnt is incremented, and the kernel can now send new ACL packets to the controller.

The acl_cnt has a similar purpose as sk_wmem_alloc field but in a lower layer.

Looking at the Grafana panels from bluetooth_exporter helped me identify that the acl_cnt was always 0 when the audio was chopping, and the time to receive a HCI_Number_Of_Completed_Packets was longer than usual. I found our bottleneck! The controller could not acknowledge the ACL packets as quickly as packets were arriving from userspace.

Knowing why this happens is tricky because the controller is a black box, which offers almost no introspectability. The hunch I have is that the 2.4Ghz is pretty noisy at some moments, and the controller needs to spend more time than usual on retransmissions and acknowledgements from the soon-to-be delivered packets. It would explain why I can only reproduce this in some days. Or another option is that it's simply a specific bug in the controller or the headphone taking too long to acknowledge the ACL packets.

A possible workaround would be to try to have a less reliable call. I found an interesting quote from the specification searching how to achieve that.

eSCO traffic should be given priority over ACL traffic in the retransmission window. – Bluetooth Core Specification Version 5.3 | Vol 2, Part B. Section 8.6.3

eSCO achieves a more reliable connection than SCO by reserving additional slots for retransmission if needed. This field is called Retransmission_Effort. Also, there is a configured value called Maximum Latency, which is the time in milliseconds it waits before giving up on the packet counting also the retransmission. Maybe I could tweak these two settings to use fewer slots and leave more ACL slots for the headphone communication.

Sniffing the HCI commands and events involved in the eSCO handshake, I stumbled upon a promising path. The HCI_Enhanced_Setup_Synchronous_Connection command configures the current parameters of an existing eSCO connection.

This command can change 23 parameters related to the current transport. I copied the same parameters and modified only the Max_Latency and Retransmission_Effort for the call. For that, I wrote a ruby script that sends a crafted hcitool cmd with the same binary data.

Unfortunately, that didn't work, and the controller replied with the following HCI event:

> HCI Event: Command Status (0x0f) plen 4             #101928 [hci0] 382.386610
      Enhanced Setup Synchronous Connection (0x01|0x003d) ncmd 1
        Status: Invalid HCI Command Parameters (0x12)

That's a bit cryptic because I had no idea what went wrong. I simply knew that the controller considers any of the 23 parameters as invalid.

Another brute force approach I thought of was to change the kernel code so these parameters could be sent directly by the kernel at the beginning of the eSCO "handshake". There is no way to set this up in userspace, so I changed the following kernel code in this section.

if (conn->pkt_type & ESCO_2EV3)
    cp.max_latency = cpu_to_le16(0x0008);
else
    // Before this was 0x000D
    cp.max_latency = cpu_to_le16(0x0008);
// Before this was a 0x002
cp.retrans_effort = 0x00;
hci_send_cmd(hdev, HCI_OP_ACCEPT_SYNC_CONN_REQ, sizeof(cp), &cp)

Trying to connect with the modified kernel didn't work because I could see that Pixel 3 wasn't accepting the eSCO negotiation.

> HCI Event: Synchronous Connect Compl.. (0x2c) plen 17  #2319 [hci0] 29.535444
        Status: Unsupported LMP Parameter Value / Unsupported LL Parameter Value (0x20)
        Handle: 0
        Address: XX:XX:XX:XX:XX:XX (Google, Inc.)
        Link type: eSCO (0x02)
        Transmission interval: 0x00
        Retransmission window: 0x00
        RX packet length: 0
        TX packet length: 0
        Air mode: Transparent (0x03)

The HFP/eSCO connection was live but downgraded to the worse CVSD codec. After that happened, I simply gave up.

I don't know if there is a bug with the closed-source controller, which would require me to buy a physical Bluetooth packet sniffer or even in the Android Bluetooth stack, which uses Bluedroid instead of BlueZ. It's better to just buy a new 15 euro Bluetooth USB stick. One communicates with the HFP/eSCO Pixel 3, and the other takes care of the A2DP/L2CAP Bose headphone.