Compare commits

...
Author SHA1 Message Date
patternihaandClaude Opus 5.5 266df8c644 TUN inbound: Document autoOutboundsInterface binding and other recent changes in README
Covers XTLS/Xray-core#7173 (a connection that cannot be bound fails
instead of leaving unbound), #7214 (IPv4-only and IPv6-only outbound
interfaces on Windows), #7230 (IPV6_BOUND_IF on macOS), and also #7130
(built-in TCP/IP stack), #6874 (IPv6 disabled system-wide on Windows),
#6811 (an existing Wintun adapter is reused) and #6883 (the iOS fd is
duplicated). #7092 and autoSystemWfpBlockLeak were already covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 23:38:34 +03:30
patternihaandClaude Opus 5.5 f2f292c90c TUN inbound: Support "tcp/port" and "udp/port" in autoSystemWfpBlockLeak on Windows; "dns" covers DoT and DoQ as well
"dns" in autoSystemWfpBlockLeak only blocked port 53 outside the TUN, plus
the DNS Client service on Windows 11 and Server 2022 and later, so other
programs' DNS over TLS (TCP 853) and DNS over QUIC (UDP 853) could still
leave outside the TUN.

autoSystemWfpBlockLeak now takes:

- "tcp/<port>", "udp/<port>": connections to that port may only go through
  the TUN (or come from Xray itself).
- "windnsclient": Windows' DNS Client service may only connect through the
  TUN, except for mDNS and LLMNR. As before, this only applies from Windows
  11 and Server 2022 on, where the service may use DoH or DoT to any port;
  earlier versions only query port 53.
- "dns": stands for "udp/53", "tcp/53", "windnsclient", "tcp/853" and
  "udp/853".
- "misconfigtun": unchanged.

Values are case-insensitive, deduplicated, and anything else, such as
"tcp/0" or "icmp/1", is rejected. Keeping the system's DNS inside the TUN,
with "udp/53", "tcp/53" or "windnsclient", needs dns to be set, as "dns"
did; ports like 853 alone do not. Each port gets its own permit-through-TUN
and block filters with a protocol condition, where port 53 used to have one
for both protocols, and the log lists what is blocked, e.g. "[tun] blocked
outside the TUN: udp/53, tcp/53, tcp/853, udp/853, windnsclient".

Tested on Windows 11 with a Wintun adapter as the outside interface: with
"tcp/853", "udp/853", "tcp/7215" and "udp/7216", those are blocked outside
the TUN over IPv4 and IPv6, while tcp/7216, udp/7215 and 443 are not, port
853 works through the TUN, Xray's own connections to port 853 still leave
outside it, and nothing is blocked after Xray stops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 14:52:11 +03:30
RPRXandGitHub 0afec66085 Xray-core v26.10.29
Sponsor & Donation & NFTs: https://github.com/XTLS/Xray-core/issues/3668
Project X Channel: https://t.me/projectXtls

Announcement of NFTs by Project X: https://github.com/XTLS/Xray-core/discussions/3633
Project X NFT: https://opensea.io/assets/ethereum/0x5ee362866001613093361eb8569d59c4141b76d1/1

VLESS Post-Quantum Encryption: https://github.com/XTLS/Xray-core/pull/5067
VLESS NFT: https://opensea.io/collection/vless

XHTTP: Beyond REALITY: https://github.com/XTLS/Xray-core/discussions/4113
REALITY NFT: https://opensea.io/assets/ethereum/0x5ee362866001613093361eb8569d59c4141b76d1/2
2026-10-29 08:38:50 +00:00
55eecaed16 Config: Move inbound module initialization to the end (#7284)
Fixes https://github.com/XTLS/Xray-core/pull/7284#issuecomment-6491545028

---------

Co-authored-by: Copilot <copilot@github.com>
2026-10-29 08:34:16 +00:00
2dfee4b842 XDRIVE transport: Carry a Google Drive session in a ring of slots (#7217)
https://github.com/XTLS/Xray-core/pull/7217#issuecomment-6489554617

---------

Co-authored-by: Ahmad-2213 <106820058+Ahmad-2213@users.noreply.github.com>
2026-10-29 06:47:42 +00:00
Жора ЗмейкинandGitHub b4123b5748 XDRIVE transport: Add "s3" storage backend (#7231)
https://github.com/XTLS/Xray-core/pull/7231#issuecomment-6488077212
2026-10-29 04:44:41 +00:00
ec124ff9e0 WebSocket transport: Treat normal close frames as io.EOF (#7236)
Fixes https://github.com/XTLS/Xray-core/issues/7235#issuecomment-6353389899

---------

Co-authored-by: 风扇滑翔翼 <Fangliding.fshxy@outlook.com>
2026-10-29 03:42:11 +00:00
SolomonandGitHub 8a3c0cbf1d Sockopt: Fix swapped TCP keepalive idle and interval on Darwin (#7219)
https://github.com/XTLS/Xray-core/pull/7219#pullrequestreview-5691960668
2026-10-29 02:54:53 +00:00
78b5a6ec5c GitHub Action CI: Enforce stricter checks on generated protobuf files (#7263)
https://github.com/XTLS/Xray-core/pull/7263#issuecomment-6447862934
https://github.com/XTLS/Xray-core/pull/7263#issuecomment-6484719865
https://github.com/XTLS/Xray-core/pull/7263#issuecomment-6485805671

---------

Co-authored-by: Fangliding <Fangliding.fshxy@outlook.com>
2026-10-29 02:34:47 +00:00
5865e3ac3d DNS outbound: Keep hijacked A/AAAA replies within the buffer (#7225)
https://github.com/XTLS/Xray-core/pull/7225#pullrequestreview-5613859264

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: 风扇滑翔翼 <Fangliding.fshxy@outlook.com>
2026-10-29 00:42:19 +00:00
48f62b801a Geodata: Limit download size, file extensions, and update frequency (#7267)
https://github.com/XTLS/Xray-core/pull/7265#issuecomment-6464588139

---------

Co-authored-by: xqzr <34030394+xqzr@users.noreply.github.com>
2026-10-29 00:22:51 +00:00
MeowandGitHub 43c29f8eb1 Geodata: Update built-in generalized "private" IP ranges (#7249)
And https://github.com/XTLS/Xray-core/pull/7249#issuecomment-6483478811

https://github.com/XTLS/Xray-core/pull/7249#issuecomment-6391494484
https://github.com/XTLS/Xray-core/pull/7249#issuecomment-6483591550
2026-10-28 23:45:22 +00:00
MeowandGitHub 613d8147bb Direct/Freedom outbound: Warn about migrating fragment and noises to Finalmask (#7242)
https://github.com/XTLS/Xray-core/issues/7240#issuecomment-6361862358
2026-10-28 22:49:30 +00:00
38f090255c Commands: Fix xray api adrules c1.json c2.json (#7276)
https://github.com/XTLS/Xray-core/pull/7276#pullrequestreview-5680949792

---------

Co-authored-by: 风扇滑翔翼 <Fangliding.fshxy@outlook.com>
2026-10-28 21:59:27 +00:00
35a3121a83 File loader: Check file extensions without resolving symlinks (#7255)
https://github.com/XTLS/Xray-core/pull/7255#issuecomment-6481194940
https://github.com/XTLS/Xray-core/pull/7250#issuecomment-6398816842

Fixes https://github.com/XTLS/Xray-core/issues/7239#issuecomment-6356122816

---------

Co-authored-by: Eljees <3.14hell@gmail.com>
2026-10-28 21:59:03 +00:00
风扇滑翔翼andGitHub bca08ac080 Routing: Fix Linux IPv6 process (#7232)
Fixes https://github.com/XTLS/Xray-core/issues/7227
2026-10-28 20:49:08 +00:00
6a4274ab75 TUN inbound: autoOutboundsInterface should not set IP_BOUND_IF on IPv6 sockets on Darwin (mainly macOS) (#7230)
Fixes https://github.com/XTLS/Xray-core/issues/7228

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 20:34:40 +00:00
solodecodeandGitHub 229799a7a3 Direct/Freedom outbound: Fix blockDelay for DispatchLink() inbounds such as VLESS, Hysteria, etc. (#7238)
https://github.com/XTLS/Xray-core/pull/7238#pullrequestreview-5627787937
https://github.com/XTLS/Xray-core/pull/7238#issuecomment-6467092152
https://github.com/XTLS/Xray-core/pull/6060#issuecomment-4384511163
2026-10-28 11:02:55 +00:00
3071fbfa25 Sockopt: Handle single-family interface binding on Windows, including TUN inbound's autoOutboundsInterface (#7214)
And https://github.com/XTLS/Xray-core/pull/7214#issuecomment-6329118637

https://github.com/XTLS/Xray-core/pull/7201#issuecomment-6314453444

Fixes https://github.com/XTLS/Xray-core/issues/7160#issuecomment-6246944012

---------

Co-authored-by: patterniha <71074308+patterniha@users.noreply.github.com>
2026-10-28 10:15:58 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1576e79563 Bump github.com/pires/go-proxyproto from 0.15.0 to 0.16.0 (#7237)
Bumps [github.com/pires/go-proxyproto](https://github.com/pires/go-proxyproto) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/pires/go-proxyproto/releases)
- [Commits](https://github.com/pires/go-proxyproto/compare/v0.15.0...v0.16.0)

---
updated-dependencies:
- dependency-name: github.com/pires/go-proxyproto
  dependency-version: 0.16.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-27 04:31:46 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
67e7f82ca1 Bump docker/login-action from 4.6.0 to 4.7.0 (#7257)
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v4.6.0...v4.7.0)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-27 04:31:31 +00:00
64 changed files with 3072 additions and 815 deletions
+1 -1
View File
@@ -82,7 +82,7 @@ jobs:
uses: docker/setup-buildx-action@v4
- name: Login to GitHub Container Registry
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
+3 -18
View File
@@ -35,24 +35,7 @@ jobs:
if: steps.check-assets.outputs.missing == 'true'
run: sleep 90
check-proto:
runs-on: ubuntu-latest
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name != github.event.pull_request.base.repo.full_name
steps:
- name: Checkout codebase
uses: actions/checkout@v7
- name: Check Proto Version Header
run: |
head -n 4 core/config.pb.go > ref.txt
find . -name "*.pb.go" ! -name "*_grpc.pb.go" -print0 | while IFS= read -r -d '' file; do
if ! cmp -s ref.txt <(head -n 4 "$file"); then
echo "Error: Header mismatch in $file"
head -n 4 "$file"
exit 1
fi
done
check-format:
check-format-and-proto:
runs-on: ubuntu-latest
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name != github.event.pull_request.base.repo.full_name
permissions:
@@ -68,6 +51,8 @@ jobs:
cache: false
- name: Check Format
run: go run ./infra/vformat/main.go -mode check -pwd ./
- name: Check Protobuf
run: go run ./infra/vprotogen/main.go -mode check -pwd ./
test:
needs: check-assets
+38 -58
View File
@@ -21,7 +21,10 @@ import (
"golang.org/x/net/http2"
)
const idleTimeout = 30 * time.Second
const (
idleTimeout = 30 * time.Second
maxDownloadSize int64 = 256 * 1024 * 1024
)
// DO NOT EDIT. Ignored by geodata, rejected by Lua.
const luaGuardHeader = "\x12\x01\x00"
@@ -33,7 +36,6 @@ type stage struct {
type downloader struct {
ctx context.Context
httpClient *http.Client
httpsClient *http.Client
}
@@ -61,12 +63,11 @@ func (c *idleConn) Write(b []byte) (int, error) {
func newDownloader(ctx context.Context, dispatcher routing.Dispatcher, outbound string) *downloader {
return &downloader{
ctx: ctx,
httpClient: newClient(ctx, dispatcher, outbound, false),
httpsClient: newClient(ctx, dispatcher, outbound, true),
httpsClient: newClient(ctx, dispatcher, outbound),
}
}
func newClient(baseCtx context.Context, dispatcher routing.Dispatcher, outbound string, isHTTPS bool) *http.Client {
func newClient(baseCtx context.Context, dispatcher routing.Dispatcher, outbound string) *http.Client {
dial := func(ctx context.Context, network, address string) (net.Conn, error) {
var conn net.Conn
err := task.Run(ctx, func() error {
@@ -91,53 +92,33 @@ func newClient(baseCtx context.Context, dispatcher routing.Dispatcher, outbound
Conn: conn,
}, nil
}
if isHTTPS {
return &http.Client{
Transport: &http2.Transport{
DialTLSContext: func(ctx context.Context, network string, address string, cfg *tls.Config) (net.Conn, error) {
conn, err := dial(ctx, network, address)
if err != nil {
return nil, err
}
host, _, _ := net.SplitHostPort(address)
tlsConn := utls.UClient(conn, &utls.Config{ServerName: host}, utls.HelloChrome_Auto)
handshakeCtx, cancel := context.WithTimeout(ctx, idleTimeout)
defer cancel()
if err := tlsConn.HandshakeContext(handshakeCtx); err != nil {
conn.Close()
return nil, err
}
return tlsConn, nil
},
},
CheckRedirect: func(req *http.Request, via []*http.Request) error {
if req.URL.Scheme != "https" {
return errors.New("redirected to non-https URL: ", req.URL.String())
return &http.Client{
Transport: &http2.Transport{
DialTLSContext: func(ctx context.Context, network string, address string, cfg *tls.Config) (net.Conn, error) {
conn, err := dial(ctx, network, address)
if err != nil {
return nil, err
}
if len(via) >= 10 {
return errors.New("stopped after 10 redirects")
host, _, _ := net.SplitHostPort(address)
tlsConn := utls.UClient(conn, &utls.Config{ServerName: host}, utls.HelloChrome_Auto)
handshakeCtx, cancel := context.WithTimeout(ctx, idleTimeout)
defer cancel()
if err := tlsConn.HandshakeContext(handshakeCtx); err != nil {
conn.Close()
return nil, err
}
return nil
return tlsConn, nil
},
}
} else {
return &http.Client{
Transport: &http.Transport{
Proxy: nil,
DisableKeepAlives: true,
DialContext: dial,
ResponseHeaderTimeout: idleTimeout,
},
CheckRedirect: func(req *http.Request, via []*http.Request) error {
if req.URL.Scheme != "https" {
return errors.New("redirected to non-https URL: ", req.URL.String())
}
if len(via) >= 10 {
return errors.New("stopped after 10 redirects")
}
return nil
},
}
},
CheckRedirect: func(req *http.Request, via []*http.Request) error {
if req.URL.Scheme != "https" {
return errors.New("redirected to non-https URL: ", req.URL.String())
}
if len(via) >= 20 {
return errors.New("stopped after 20 redirects")
}
return nil
},
}
}
@@ -203,27 +184,26 @@ func (d *downloader) fetch(rawURL string, writer io.Writer) error {
}
utils.TryDefaultHeadersWith(req.Header, "nav")
var client *http.Client
if req.URL.Scheme == "https" {
client = d.httpsClient
} else {
client = d.httpClient
}
resp, err := client.Do(req)
resp, err := d.httpsClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
io.Copy(io.Discard, resp.Body)
return errors.New("unexpected status code: ", resp.StatusCode)
}
if resp.ContentLength > maxDownloadSize {
return errors.New("geodata download exceeds 256 MiB")
}
n, err := io.Copy(writer, resp.Body)
n, err := io.Copy(writer, io.LimitReader(resp.Body, maxDownloadSize+1))
if err != nil {
return err
}
if n > maxDownloadSize {
return errors.New("geodata download exceeds 256 MiB")
}
if n == 0 {
return errors.New("empty response body")
}
+55 -2
View File
@@ -7,10 +7,13 @@ import (
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
commongeodata "github.com/xtls/xray-core/common/geodata"
xlua "github.com/xtls/xray-core/common/lua"
"github.com/xtls/xray-core/common/serial"
"github.com/xtls/xray-core/core"
"github.com/yuin/gopher-lua/parse"
"google.golang.org/protobuf/proto"
)
@@ -31,11 +34,11 @@ func TestDownloadOneLuaGuard(t *testing.T) {
if err := os.WriteFile(filepath.Join(dir, "asset.lua"), nil, 0o600); err != nil {
t.Fatal(err)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(tc.body))
}))
defer server.Close()
d := &downloader{ctx: context.Background(), httpClient: server.Client()}
d := &downloader{ctx: context.Background(), httpsClient: server.Client()}
staged, err := d.downloadOne(&Asset{Url: server.URL, File: "asset.lua"})
if err != nil {
t.Fatal(err)
@@ -71,3 +74,53 @@ func TestDownloadOneLuaGuard(t *testing.T) {
})
}
}
func TestProtobufConfigValidation(t *testing.T) {
dir := t.TempDir()
t.Setenv("xray.location.asset", dir)
for _, name := range []string{"asset.dat", "asset.DAT"} {
if err := os.WriteFile(filepath.Join(dir, name), nil, 0o600); err != nil {
t.Fatal(err)
}
}
if err := os.Mkdir(filepath.Join(dir, "directory.dat"), 0o700); err != nil {
t.Fatal(err)
}
server, err := core.New(&core.Config{})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = server.Close() })
for _, tc := range []struct {
name string
config *Config
wantError string
}{
{"minutely cron", &Config{Cron: "* * * * *"}, ""},
{"minimum interval", &Config{Cron: "@every 1m"}, ""},
{"short interval", &Config{Cron: "@every 59s"}, "interval must be at least 1 minute"},
{"fractional interval", &Config{Cron: "@every 59.999s"}, "interval must be at least 1 minute"},
{"invalid cron", &Config{Cron: "invalid"}, "invalid geodata cron"},
{"dat extension", &Config{Cron: "@daily", Assets: []*Asset{{Url: "https://example.com/asset.dat", File: "asset.dat"}}}, ""},
{"uppercase extension", &Config{Cron: "@daily", Assets: []*Asset{{Url: "https://example.com/asset.dat", File: "asset.DAT"}}}, ""},
{"invalid extension", &Config{Cron: "@daily", Assets: []*Asset{{Url: "https://example.com/asset.dat", File: "asset.lua"}}}, "extension must be .dat"},
{"missing extension", &Config{Cron: "@daily", Assets: []*Asset{{Url: "https://example.com/asset.dat", File: "asset"}}}, "extension must be .dat"},
{"missing file", &Config{Cron: "@daily", Assets: []*Asset{{Url: "https://example.com/asset.dat", File: "missing.dat"}}}, "invalid geodata asset file"},
{"directory", &Config{Cron: "@daily", Assets: []*Asset{{Url: "https://example.com/asset.dat", File: "directory.dat"}}}, "asset is not a regular file"},
} {
t.Run(tc.name, func(t *testing.T) {
config, err := serial.ToTypedMessage(tc.config).GetInstance()
if err != nil {
t.Fatal(err)
}
_, err = core.CreateObject(server, config)
if tc.wantError == "" {
if err != nil {
t.Fatal(err)
}
} else if err == nil || !strings.Contains(err.Error(), tc.wantError) {
t.Fatalf("CreateObject() error = %v; want %q", err, tc.wantError)
}
})
}
}
+23 -3
View File
@@ -2,12 +2,16 @@ package geodata
import (
"context"
"path/filepath"
"strings"
"sync"
"time"
"github.com/robfig/cron/v3"
"github.com/xtls/xray-core/common"
"github.com/xtls/xray-core/common/errors"
commongeodata "github.com/xtls/xray-core/common/geodata"
"github.com/xtls/xray-core/common/platform/filesystem"
"github.com/xtls/xray-core/core"
"github.com/xtls/xray-core/features/routing"
)
@@ -25,6 +29,24 @@ func New(ctx context.Context, config *Config) (*Instance, error) {
if config.Cron == "" {
return &Instance{}, nil
}
if len(config.Assets) > 100 {
return nil, errors.New("geodata supports at most 100 assets")
}
for _, asset := range config.Assets {
if !strings.EqualFold(filepath.Ext(asset.GetFile()), ".dat") {
return nil, errors.New("geodata asset file extension must be .dat")
}
if _, err := filesystem.ResolveAsset(asset.GetFile()); err != nil {
return nil, errors.New("invalid geodata asset file: ", asset.GetFile()).Base(err)
}
}
schedule, err := cron.ParseStandard(config.Cron)
if err != nil {
return nil, errors.New("invalid geodata cron").Base(err)
}
if interval, ok := schedule.(cron.ConstantDelaySchedule); ok && interval.Delay < time.Minute {
return nil, errors.New("geodata cron interval must be at least 1 minute")
}
g := &Instance{
assets: config.Assets,
@@ -44,9 +66,7 @@ func New(ctx context.Context, config *Config) (*Instance, error) {
cron.WithChain(cron.SkipIfStillRunning(cron.DiscardLogger)),
cron.WithLogger(cron.DiscardLogger),
)
if _, err := g.tasker.AddFunc(config.Cron, g.execute); err != nil {
return nil, errors.New("invalid geodata cron").Base(err)
}
g.tasker.Schedule(schedule, cron.FuncJob(g.execute))
errors.LogInfo(ctx, "scheduled geodata reload with cron: ", config.Cron)
return g, nil
+12 -12
View File
@@ -152,23 +152,23 @@ func (x *HealthPingMeasurementResult) GetMin() int64 {
type OutboundStatus struct {
state protoimpl.MessageState `protogen:"open.v1"`
// @Document Whether this outbound is usable
// @Restriction ReadOnlyForUser
//@Restriction ReadOnlyForUser
Alive bool `protobuf:"varint,1,opt,name=alive,proto3" json:"alive,omitempty"`
// @Document The time for probe request to finish.
// @Type time.ms
// @Restriction ReadOnlyForUser
//@Type time.ms
//@Restriction ReadOnlyForUser
Delay int64 `protobuf:"varint,2,opt,name=delay,proto3" json:"delay,omitempty"`
// @Document The last error caused this outbound failed to relay probe request
// @Restriction NotMachineReadable
//@Restriction NotMachineReadable
LastErrorReason string `protobuf:"bytes,3,opt,name=last_error_reason,json=lastErrorReason,proto3" json:"last_error_reason,omitempty"`
// @Document The outbound tag for this Server
// @Type id.outboundTag
//@Type id.outboundTag
OutboundTag string `protobuf:"bytes,4,opt,name=outbound_tag,json=outboundTag,proto3" json:"outbound_tag,omitempty"`
// @Document The time this outbound is known to be alive
// @Type id.outboundTag
//@Type id.outboundTag
LastSeenTime int64 `protobuf:"varint,5,opt,name=last_seen_time,json=lastSeenTime,proto3" json:"last_seen_time,omitempty"`
// @Document The time this outbound is tried
// @Type id.outboundTag
//@Type id.outboundTag
LastTryTime int64 `protobuf:"varint,6,opt,name=last_try_time,json=lastTryTime,proto3" json:"last_try_time,omitempty"`
HealthPing *HealthPingMeasurementResult `protobuf:"bytes,7,opt,name=health_ping,json=healthPing,proto3" json:"health_ping,omitempty"`
unknownFields protoimpl.UnknownFields
@@ -257,14 +257,14 @@ func (x *OutboundStatus) GetHealthPing() *HealthPingMeasurementResult {
type ProbeResult struct {
state protoimpl.MessageState `protogen:"open.v1"`
// @Document Whether this outbound is usable
// @Restriction ReadOnlyForUser
//@Restriction ReadOnlyForUser
Alive bool `protobuf:"varint,1,opt,name=alive,proto3" json:"alive,omitempty"`
// @Document The time for probe request to finish.
// @Type time.ms
// @Restriction ReadOnlyForUser
//@Type time.ms
//@Restriction ReadOnlyForUser
Delay int64 `protobuf:"varint,2,opt,name=delay,proto3" json:"delay,omitempty"`
// @Document The error caused this outbound failed to relay probe request
// @Restriction NotMachineReadable
//@Restriction NotMachineReadable
LastErrorReason string `protobuf:"bytes,3,opt,name=last_error_reason,json=lastErrorReason,proto3" json:"last_error_reason,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
@@ -324,7 +324,7 @@ func (x *ProbeResult) GetLastErrorReason() string {
type Intensity struct {
state protoimpl.MessageState `protogen:"open.v1"`
// @Document The time interval for a probe request in ms.
// @Type time.ms
//@Type time.ms
ProbeInterval uint32 `protobuf:"varint,1,opt,name=probe_interval,json=probeInterval,proto3" json:"probe_interval,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
+13 -12
View File
@@ -34,18 +34,19 @@ message RoutingContext {
// opened by xray-core.
// * FieldSelectors selects a subset of fields in routing statistics to return.
// Valid selectors:
// - inbound: Selects connection's inbound tag.
// - network: Selects connection's network.
// - ip: Equivalent as "ip_source" and "ip_target", selects both source and
// target IP.
// - port: Equivalent as "port_source" and "port_target", selects both source
// and target port.
// - domain: Selects target domain.
// - protocol: Select connection's protocol.
// - user: Select connection's inbound user email.
// - attributes: Select connection's additional attributes.
// - outbound: Equivalent as "outbound" and "outbound_group", select both
// outbound tag and outbound group tags.
// - inbound: Selects connection's inbound tag.
// - network: Selects connection's network.
// - ip: Equivalent as "ip_source" and "ip_target", selects both source and
// target IP.
// - port: Equivalent as "port_source" and "port_target", selects both source
// and target port.
// - domain: Selects target domain.
// - protocol: Select connection's protocol.
// - user: Select connection's inbound user email.
// - attributes: Select connection's additional attributes.
// - outbound: Equivalent as "outbound" and "outbound_group", select both
// outbound tag and outbound group tags.
//
// * If FieldSelectors is left empty, all fields will be returned.
message SubscribeRoutingStatsRequest {
repeated string FieldSelectors = 1;
+59 -17
View File
@@ -8,23 +8,65 @@ import (
var privateIPMatcher = sync.OnceValue(func() IPMatcher {
return common.Must2(IPReg.BuildIPMatcher(common.Must2(ParseIPRules([]string{
"0.0.0.0/8",
"10.0.0.0/8",
"100.64.0.0/10",
"127.0.0.0/8",
"169.254.0.0/16",
"172.16.0.0/12",
"192.0.0.0/24",
"192.0.2.0/24",
"192.88.99.0/24",
"192.168.0.0/16",
"198.18.0.0/15",
"198.51.100.0/24",
"203.0.113.0/24",
"224.0.0.0/3",
"::/127",
"fc00::/7",
"fe80::/10",
// https://www.iana.org/assignments/iana-ipv4-special-registry
"0.0.0.0/8", // "This network"
"0.0.0.0/32", // "This host on this network"
"10.0.0.0/8", // Private-Use
"100.64.0.0/10", // Shared Address Space
"127.0.0.0/8", // Loopback
"169.254.0.0/16", // Link Local
"172.16.0.0/12", // Private-Use
// "192.0.0.0/24", // IETF Protocol Assignments
"192.0.0.0/29", // IPv4 Service Continuity Prefix
"192.0.0.8/32", // IPv4 dummy address
"192.0.0.9/32", // Port Control Protocol Anycast
"192.0.0.10/32", // Traversal Using Relays around NAT Anycast
"192.0.0.170/32", // NAT64/DNS64 Discovery
"192.0.0.171/32", // NAT64/DNS64 Discovery
"192.0.2.0/24", // Documentation (TEST-NET-1)
// "192.31.196.0/24", // AS112-v4
"192.52.193.0/24", // AMT
// "192.88.99.0/24", // Deprecated (6to4 Relay Anycast)
"192.88.99.2/32", // 6a44-relay anycast address
"192.168.0.0/16", // Private-Use
// "192.175.48.0/24", // Direct Delegation AS112 Service
"198.18.0.0/15", // Benchmarking
"198.51.100.0/24", // Documentation (TEST-NET-2)
"203.0.113.0/24", // Documentation (TEST-NET-3)
"240.0.0.0/4", // Reserved
"255.255.255.255/32", // Limited Broadcast
// https://www.iana.org/assignments/multicast-addresses
"224.0.0.0/4",
// https://www.iana.org/assignments/iana-ipv6-special-registry
"::1/128", // Loopback Address
"::/128", // Unspecified Address
// "::ffff:0:0/96", // IPv4-mapped Address
"64:ff9b::/96", // IPv4-IPv6 Translat.
"64:ff9b:1::/48", // IPv4-IPv6 Translat.
"100::/64", // Discard-Only Address Block
"100:0:0:1::/64", // Dummy IPv6 Prefix
// "2001::/23", // IETF Protocol Assignments
"2001::/32", // TEREDO
"2001:1::1/128", // Port Control Protocol Anycast
"2001:1::2/128", // Traversal Using Relays around NAT Anycast
"2001:1::3/128", // DNS-SD Service Registration Protocol Anycast
"2001:2::/48", // Benchmarking
"2001:3::/32", // AMT
// "2001:4:112::/48", // AS112-v6
// "2001:10::/28", // Deprecated (previously ORCHID)
"2001:20::/28", // ORCHIDv2
"2001:30::/28", // Drone Remote ID Protocol Entity Tags (DETs) Prefix
"2001:db8::/32", // Documentation
"2002::/16", // 6to4
// "2620:4f:8000::/48", // Direct Delegation AS112 Service
"3fff::/20", // Documentation
"5f00::/16", // Segment Routing (SRv6) SIDs
"fc00::/7", // Unique-Local
"fe80::/10", // Link-Local Unicast
// https://www.iana.org/assignments/ipv6-multicast-addresses
"ff00::/8",
}))))
})
+2 -2
View File
@@ -11,7 +11,7 @@ import (
"github.com/xtls/xray-core/common/base64"
"github.com/xtls/xray-core/common/errors"
"github.com/xtls/xray-core/common/platform"
"github.com/xtls/xray-core/common/platform/filesystem"
glua "github.com/yuin/gopher-lua"
"github.com/yuin/gopher-lua/parse"
)
@@ -36,7 +36,7 @@ func Compile(script string) (*Program, error) {
// TODO: Restore inline script support and re-enable the related tests.
return nil, errors.New("currently unavailable")
code, err = base64.Decode(script)
} else if name, err = platform.ResolveLuaFile(script); err == nil {
} else if name, err = filesystem.ResolveLuaFile(script); err == nil {
code, err = os.ReadFile(name)
// TODO: Restore privileged signature authorization and re-enable the related tests and CLI entries.
// if err == nil {
+8 -11
View File
@@ -187,19 +187,20 @@ func TestCompileInlineDiagnostics(t *testing.T) {
*/
func TestCompileFileExtension(t *testing.T) {
for _, name := range []string{"script", "script.dat", "script.lua.txt", "script.LUA"} {
for _, name := range []string{"script", "script.dat", "script.lua.txt"} {
if program, err := Compile(name); program != nil || err == nil || !strings.Contains(err.Error(), "Lua file extension must be .lua") {
t.Fatalf("Compile(%q) = %v, %v; want nil program and .lua extension error", name, program, err)
}
}
for _, name := range []string{"script.lua", "script.LUA", "script.Lua", "script.lUa"} {
path := filepath.Join(t.TempDir(), name)
if err := os.WriteFile(path, []byte("value = 42"), 0o600); err != nil {
t.Fatal(err)
}
if program, err := Compile(path); program != nil || err == nil || !strings.Contains(err.Error(), "Lua file extension must be .lua") {
t.Fatalf("Compile(%q) = %v, %v; want nil program and .lua extension error", path, program, err)
if program, err := Compile(path); program == nil || err != nil {
t.Fatalf("Compile(%q) = %v, %v; want successful compilation", path, program, err)
}
}
path := writeTestScript(t, "value = 42") + string(filepath.Separator) + "."
if program, err := Compile(path); program == nil || err != nil {
t.Fatalf("Compile(%q) = %v, %v; want acceptance of the resolved .lua path", path, program, err)
}
}
func TestCompileRejectsReadFailure(t *testing.T) {
@@ -234,10 +235,6 @@ func writeTestScript(t *testing.T, code string) string {
if err := os.WriteFile(path, []byte(code), 0o600); err != nil {
t.Fatal(err)
}
path, err := filepath.EvalSymlinks(path)
if err != nil {
t.Fatal(err)
}
return path
}
+12 -18
View File
@@ -4,7 +4,7 @@ package net
import (
"bufio"
"encoding/hex"
"encoding/binary"
"fmt"
"net"
"os"
@@ -45,10 +45,7 @@ func FindProcess(network, srcIP string, srcPort uint16, destIP string, destPort
panic("Unsupported network type for process lookup.")
}
targetHexAddr, err := formatLittleEndianString(net.ParseIP(srcIP), Port(srcPort))
if err != nil {
return 0, "", "", errors.New("failed to format address: ", err)
}
targetHexAddr := formatProcNetAddress(net.ParseIP(srcIP), Port(srcPort))
portFallback := ""
if network == "udp" {
@@ -87,24 +84,21 @@ func FindProcess(network, srcIP string, srcPort uint16, destIP string, destPort
return pid, procName, absPath, nil
}
func formatLittleEndianString(addr net.IP, port Port) (string, error) {
ip := addr
func formatProcNetAddress(addr net.IP, port Port) string {
var ipBytes []byte
if ip.To4() != nil {
ipBytes = ip.To4()
if ipv4 := addr.To4(); ipv4 != nil {
ipBytes = ipv4
} else {
ipBytes = ip.To16()
}
if ipBytes == nil {
return "", errors.New("invalid IP format for ", addr, ": ", ip)
ipBytes = addr.To16()
}
for i, j := 0, len(ipBytes)-1; i < j; i, j = i+1, j-1 {
ipBytes[i], ipBytes[j] = ipBytes[j], ipBytes[i]
var formatted strings.Builder
formatted.Grow(len(ipBytes)*2 + 5)
for i := 0; i < len(ipBytes); i += 4 {
fmt.Fprintf(&formatted, "%08X", binary.NativeEndian.Uint32(ipBytes[i:i+4]))
}
portHex := fmt.Sprintf("%04X", uint16(port))
ipHex := strings.ToUpper(hex.EncodeToString(ipBytes))
return fmt.Sprintf("%s:%s", ipHex, portHex), nil
fmt.Fprintf(&formatted, ":%04X", uint16(port))
return formatted.String()
}
func findInodeInFiles(filePaths []string, targetHexAddr, portFallback string) (string, error) {
+24 -24
View File
@@ -5,6 +5,7 @@ import (
"io"
"os"
"path/filepath"
"strings"
"github.com/xtls/xray-core/common/buf"
"github.com/xtls/xray-core/common/platform"
@@ -26,14 +27,6 @@ func ReadFile(path string) ([]byte, error) {
return buf.ReadAllToBytes(reader)
}
func ReadAsset(file string) ([]byte, error) {
path, _, err := getAssetFileLocation(file)
if err != nil {
return nil, err
}
return ReadFile(path)
}
func OpenAsset(file string) (io.ReadCloser, int64, error) {
path, info, err := getAssetFileLocation(file)
if err != nil {
@@ -43,11 +36,6 @@ func OpenAsset(file string) (io.ReadCloser, int64, error) {
return r, info.Size(), err
}
func StatAsset(file string) (os.FileInfo, error) {
_, info, err := getAssetFileLocation(file)
return info, err
}
func ResolveAsset(file string) (string, error) {
path, _, err := getAssetFileLocation(file)
return path, err
@@ -79,17 +67,29 @@ func ReadCert(file string) ([]byte, error) {
return ReadFile(platform.GetCertLocation(file))
}
func CopyFile(dst string, src string) error {
bytes, err := ReadFile(src)
if err != nil {
return err
// ResolveLuaFile finds a local .lua script and returns its absolute path.
// Relative paths: XRAY_LOCATION_CONFDIR > XRAY_LOCATION_CONFIG > working dir > executable dir.
func ResolveLuaFile(path string) (string, error) {
if !strings.EqualFold(filepath.Ext(path), ".lua") {
return "", errors.New("Lua file extension must be .lua")
}
f, err := os.OpenFile(dst, os.O_CREATE|os.O_WRONLY, 0o644)
if err != nil {
return err
if filepath.IsAbs(path) {
return path, nil
}
defer f.Close()
_, err = f.Write(bytes)
return err
for _, dir := range []string{
platform.GetConfDirPath(),
platform.NewEnvFlag(platform.ConfigLocation).GetValue(func() string { return "" }),
".",
platform.GetExecutableDir(),
} {
if dir != "" {
resolved := filepath.Join(dir, path)
if _, err := os.Stat(resolved); err == nil {
return filepath.Abs(resolved)
} else if !os.IsNotExist(err) {
return "", err
}
}
}
return "", os.ErrNotExist
}
+47 -2
View File
@@ -1,13 +1,17 @@
package filesystem_test
import (
"errors"
"os"
"path/filepath"
"testing"
"github.com/xtls/xray-core/common"
"github.com/xtls/xray-core/common/platform"
. "github.com/xtls/xray-core/common/platform/filesystem"
)
func TestStatAssetRejectsInvalidPath(t *testing.T) {
func TestResolveAssetRejectsInvalidPath(t *testing.T) {
for _, file := range []string{
"",
".",
@@ -25,8 +29,49 @@ func TestStatAssetRejectsInvalidPath(t *testing.T) {
`nested\..\geoip.dat`,
filepath.Join(t.TempDir(), "geoip.dat"),
} {
if _, err := StatAsset(file); err == nil {
if _, err := ResolveAsset(file); err == nil {
t.Fatalf("expected error for %q", file)
}
}
}
func TestResolveLuaFile(t *testing.T) {
t.Chdir(t.TempDir())
workingDir, err := os.Getwd()
common.Must(err)
executable, err := os.Executable()
common.Must(err)
file, err := os.CreateTemp(filepath.Dir(executable), "lua-*.lua")
common.Must(err)
common.Must(file.Close())
defer os.Remove(file.Name())
name := filepath.Base(file.Name())
paths := []string{
filepath.Join(t.TempDir(), name),
filepath.Join(t.TempDir(), name),
filepath.Join(workingDir, name),
file.Name(),
}
t.Setenv(platform.ConfdirLocation, filepath.Dir(paths[0]))
t.Setenv(platform.ConfigLocation, filepath.Dir(paths[1]))
for _, path := range paths[:3] {
common.Must(os.WriteFile(path, nil, 0o600))
}
if got, err := ResolveLuaFile(paths[2]); err != nil || got != paths[2] {
t.Fatalf("absolute path = %q, %v; want %q", got, err, paths[2])
}
for i, want := range paths {
if i == 2 {
t.Setenv(platform.ConfdirLocation, "")
t.Setenv(platform.ConfigLocation, "")
}
if got, err := ResolveLuaFile(name); err != nil || got != want {
t.Fatalf("resolved path = %q, %v; want %q", got, err, want)
}
common.Must(os.Remove(want))
}
if _, err := ResolveLuaFile(name); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("missing file error = %v", err)
}
}
+2 -2
View File
@@ -14,7 +14,7 @@ func LineSeparator() string {
// GetAssetLocation searches for `file` in the env dir, the executable dir, and certain locations
func GetAssetLocation(file string) string {
assetPath := NewEnvFlag(AssetLocation).GetValue(getExecutableDir)
assetPath := NewEnvFlag(AssetLocation).GetValue(GetExecutableDir)
defPath := filepath.Join(assetPath, file)
for _, p := range []string{
defPath,
@@ -36,6 +36,6 @@ func GetAssetLocation(file string) string {
// GetCertLocation searches for `file` in the env dir and the executable dir
func GetCertLocation(file string) string {
certPath := NewEnvFlag(CertLocation).GetValue(getExecutableDir)
certPath := NewEnvFlag(CertLocation).GetValue(GetExecutableDir)
return filepath.Join(certPath, file)
}
+2 -65
View File
@@ -1,8 +1,6 @@
package platform // import "github.com/xtls/xray-core/common/platform"
import (
"errors"
"fmt"
"os"
"path/filepath"
"strconv"
@@ -74,7 +72,7 @@ func NormalizeEnvName(name string) string {
return strings.ReplaceAll(strings.ToUpper(strings.TrimSpace(name)), ".", "_")
}
func getExecutableDir() string {
func GetExecutableDir() string {
exec, err := os.Executable()
if err != nil {
return ""
@@ -83,7 +81,7 @@ func getExecutableDir() string {
}
func GetConfigurationPath() string {
configPath := NewEnvFlag(ConfigLocation).GetValue(getExecutableDir)
configPath := NewEnvFlag(ConfigLocation).GetValue(GetExecutableDir)
return filepath.Join(configPath, "config.json")
}
@@ -92,64 +90,3 @@ func GetConfDirPath() string {
configPath := NewEnvFlag(ConfdirLocation).GetValue(func() string { return "" })
return configPath
}
// ResolveLuaFile finds a local .lua script and returns its resolved absolute path.
// Relative paths: XRAY_LOCATION_CONFDIR > XRAY_LOCATION_CONFIG > working dir > executable dir.
func ResolveLuaFile(path string) (string, error) {
if path == "" {
return "", errors.New("Lua file path is empty")
}
paths := []string{path}
if !filepath.IsAbs(path) {
paths = nil
for _, dir := range []string{
GetConfDirPath(),
NewEnvFlag(ConfigLocation).GetValue(func() string { return "" }),
".",
getExecutableDir(),
} {
if dir != "" {
paths = append(paths, filepath.Join(dir, path))
}
}
}
path, err := resolveFile(paths)
if err != nil {
return "", err
}
return resolveLuaTarget(path)
}
func resolveLuaTarget(path string) (string, error) {
path, err := filepath.EvalSymlinks(path)
if err != nil {
return "", fmt.Errorf("failed to resolve Lua file symlinks: %w", err)
}
if filepath.Ext(path) != ".lua" {
return "", errors.New("Lua file extension must be .lua")
}
return path, nil
}
func resolveFile(paths []string) (string, error) {
var tried []string
for _, path := range paths {
path, err := filepath.Abs(path)
if err != nil {
return "", fmt.Errorf("failed to resolve file path: %w", err)
}
tried = append(tried, path)
info, err := os.Stat(path)
if errors.Is(err, os.ErrNotExist) {
continue
}
if err != nil {
return "", fmt.Errorf("failed to inspect file %q: %w", path, err)
}
if !info.Mode().IsRegular() {
return "", fmt.Errorf("file is not a regular file: %s", path)
}
return path, nil
}
return "", fmt.Errorf("file not found; tried %q: %w", tried, os.ErrNotExist)
}
-88
View File
@@ -1,11 +1,9 @@
package platform_test
import (
"errors"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"github.com/xtls/xray-core/common"
@@ -66,89 +64,3 @@ func TestGetAssetLocation(t *testing.T) {
}
}
}
func TestResolveLuaFile(t *testing.T) {
workingDir := t.TempDir()
t.Chdir(workingDir)
executable, err := os.Executable()
common.Must(err)
file, err := os.CreateTemp(filepath.Dir(executable), "lua-*.lua")
common.Must(err)
common.Must(file.Close())
defer os.Remove(file.Name())
name := filepath.Base(file.Name())
paths := []string{
filepath.Join(t.TempDir(), name),
filepath.Join(t.TempDir(), name),
filepath.Join(workingDir, name),
file.Name(),
}
t.Setenv(ConfdirLocation, filepath.Dir(paths[0]))
t.Setenv(ConfigLocation, filepath.Dir(paths[1]))
for _, path := range paths[:3] {
common.Must(os.WriteFile(path, nil, 0o600))
}
for i, path := range paths {
paths[i], err = filepath.EvalSymlinks(path)
common.Must(err)
}
if got, err := ResolveLuaFile(paths[2]); err != nil || got != paths[2] {
t.Fatalf("absolute path = %q, %v; want %q", got, err, paths[2])
}
for i, want := range paths {
if i == 2 {
t.Setenv(ConfdirLocation, "")
t.Setenv(ConfigLocation, "")
}
if got, err := ResolveLuaFile(name); err != nil || got != want {
t.Fatalf("resolved path = %q, %v; want %q", got, err, want)
}
common.Must(os.Remove(want))
}
if _, err := ResolveLuaFile(name); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("missing file error = %v", err)
}
t.Setenv(ConfdirLocation, filepath.Dir(paths[0]))
t.Setenv(ConfigLocation, filepath.Dir(paths[1]))
common.Must(os.Mkdir(paths[0], 0o700))
common.Must(os.WriteFile(paths[1], nil, 0o600))
for _, path := range []string{"", name, filepath.Join(t.TempDir(), name)} {
if _, err := ResolveLuaFile(path); err == nil {
t.Fatalf("accepted invalid path %q", path)
}
}
}
func TestResolveLuaFileSymlink(t *testing.T) {
for _, tc := range []struct {
link, target string
valid bool
}{
{"alias.lua", "data.dat", false},
{"alias.dat", "script.lua", true},
} {
t.Run(tc.link, func(t *testing.T) {
dir, err := filepath.EvalSymlinks(t.TempDir())
common.Must(err)
target := filepath.Join(dir, tc.target)
common.Must(os.WriteFile(target, nil, 0o600))
link := filepath.Join(dir, tc.link)
if err := os.Symlink(tc.target, link); err != nil {
if runtime.GOOS == "windows" {
t.Skipf("cannot create symlinks: %v", err)
}
t.Fatal(err)
}
got, err := ResolveLuaFile(link)
if tc.valid {
if err != nil || got != target {
t.Fatalf("ResolveLuaFile(%q) = %q, %v; want %q", link, got, err, target)
}
} else if err == nil || got != "" || !strings.Contains(err.Error(), "Lua file extension must be .lua") {
t.Fatalf("ResolveLuaFile(%q) = %q, %v; want .lua extension error", link, got, err)
}
})
}
}
+2 -2
View File
@@ -11,12 +11,12 @@ func LineSeparator() string {
// GetAssetLocation searches for `file` in the env dir and the executable dir
func GetAssetLocation(file string) string {
assetPath := NewEnvFlag(AssetLocation).GetValue(getExecutableDir)
assetPath := NewEnvFlag(AssetLocation).GetValue(GetExecutableDir)
return filepath.Join(assetPath, file)
}
// GetCertLocation searches for `file` in the env dir and the executable dir
func GetCertLocation(file string) string {
certPath := NewEnvFlag(CertLocation).GetValue(getExecutableDir)
certPath := NewEnvFlag(CertLocation).GetValue(GetExecutableDir)
return filepath.Join(certPath, file)
}
+1 -1
View File
@@ -20,7 +20,7 @@ import (
var (
Version_x byte = 26
Version_y byte = 10
Version_z byte = 21
Version_z byte = 29
)
var (
-2
View File
@@ -1,5 +1,3 @@
package core
//go:generate go install -v google.golang.org/protobuf/cmd/protoc-gen-go@latest
//go:generate go install -v google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
//go:generate go run ../infra/vprotogen/main.go -pwd ./..
+10
View File
@@ -2,6 +2,7 @@ package localdns
import (
"context"
"net/netip"
"syscall"
"time"
@@ -100,6 +101,15 @@ func New() *Client {
if internet.IsSkippedDNSServer(address) {
return nil, errors.New("skipped DNS server ", address)
}
// A socket of the server's IP version only needs that version bound
// to the interface, which a dual-stack one would need both of.
if ap, err := netip.ParseAddrPort(address); err == nil && (network == "udp" || network == "tcp") {
if ap.Addr().Unmap().Is4() {
network += "4"
} else {
network += "6"
}
}
return d.DialContext(ctx, network, address)
},
}
+9 -2
View File
@@ -4,6 +4,7 @@ go 1.27
require (
github.com/apernet/quic-go v0.61.1-0.20260806010916-184d081eef3e
github.com/bufbuild/protocompile v0.14.1
github.com/ghodss/yaml v1.0.1-0.20220118164431-d8423dcdf344
github.com/golang/mock v1.7.0-rc.1
github.com/google/go-cmp v0.7.0
@@ -14,7 +15,7 @@ require (
github.com/miekg/dns v1.1.73
github.com/pelletier/go-toml v1.9.5
github.com/pion/stun/v3 v3.1.7
github.com/pires/go-proxyproto v0.15.0
github.com/pires/go-proxyproto v0.16.0
github.com/refraction-networking/utls v1.8.3-0.20260301010127-aa6edf4b11af
github.com/robfig/cron/v3 v3.0.1
github.com/stretchr/testify v1.12.1
@@ -32,7 +33,7 @@ require (
golang.zx2c4.com/wireguard v0.0.0-20250521234502-f333402bd9cb
golang.zx2c4.com/wireguard/windows v1.1.1
google.golang.org/grpc v1.84.0
google.golang.org/protobuf v1.36.12
google.golang.org/protobuf v1.36.11
h12.io/socks v1.0.3
layeh.com/gopher-luar v1.0.11
lukechampine.com/blake3 v1.4.1
@@ -58,6 +59,12 @@ require (
golang.org/x/text v0.43.0 // indirect
golang.org/x/tools v0.51.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260706201446-f0a921348800 // indirect
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.6.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gvisor.dev/gvisor v0.0.0-20260122175437-89a5d21be8f0 // indirect
)
tool (
google.golang.org/grpc/cmd/protoc-gen-go-grpc
google.golang.org/protobuf/cmd/protoc-gen-go
)
+8 -2
View File
@@ -2,6 +2,8 @@ github.com/andybalholm/brotli v1.2.3 h1:8H1qwOkl2LPfjf3YezB90JnCliZb6SInJ/OJkEbA
github.com/andybalholm/brotli v1.2.3/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/apernet/quic-go v0.61.1-0.20260806010916-184d081eef3e h1:5mgtR5gwIgBKMiGI1QdXldZZ+SNor06Nbu1wCBulQBg=
github.com/apernet/quic-go v0.61.1-0.20260806010916-184d081eef3e/go.mod h1:x7qxEvX6MCVtDuBKHj3E+88+BtrbEMuAL5qGUKItjW8=
github.com/bufbuild/protocompile v0.14.1 h1:iA73zAf/fyljNjQKwYzUHD6AD4R8KMasmwa/FBatYVw=
github.com/bufbuild/protocompile v0.14.1/go.mod h1:ppVdAIhbr2H8asPk6k4pY7t9zB1OU5DoEw9xY/FUi1c=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
@@ -59,8 +61,8 @@ github.com/pion/stun/v3 v3.1.7 h1:uRXMTlGLf89WgItGNyZ6aR5jMTX0NBbybXADpQCzn+E=
github.com/pion/stun/v3 v3.1.7/go.mod h1:Nq77RW4aRrSNrltf2ksUJLjxWeipj4lnlgdsYIxC8g8=
github.com/pion/transport/v4 v4.1.0 h1:8S+nF2reM2cJuqC6g78OVy2BBgmbdns+acx3jA97BvQ=
github.com/pion/transport/v4 v4.1.0/go.mod h1:06hFI+jCFcok2X2MekVufNZ/uzNZXivGBPfviSVcjgM=
github.com/pires/go-proxyproto v0.15.0 h1:dTshmNbFm/D+0+sbrxUuddPOZ5Y0B7c5NhtsBkm6LqI=
github.com/pires/go-proxyproto v0.15.0/go.mod h1:OXsCrKwrK2tXS9YrI5tkHx5xaQlO8FH3lFW76orFh24=
github.com/pires/go-proxyproto v0.16.0 h1:9KYT/fM3wpjT8bdMEj/7FBcoiUG1KyzEjRYnPu+vz7A=
github.com/pires/go-proxyproto v0.16.0/go.mod h1:n7yhg8yIfNl3KgYDCwBzmeUHNpKTVnUhcWpc7rV4RW4=
github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0=
github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk=
github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
@@ -152,6 +154,10 @@ google.golang.org/genproto/googleapis/rpc v0.0.0-20260706201446-f0a921348800 h1:
google.golang.org/genproto/googleapis/rpc v0.0.0-20260706201446-f0a921348800/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.84.0 h1:soMyaPJ8pAak5PIQ0DGBUir0XRo2fRoMqhNWMLlLxO0=
google.golang.org/grpc v1.84.0/go.mod h1:ljCht0DrxQrXBDRTZp52Qxh3Ffk8CdYm2sj4O2QN2C0=
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.6.0 h1:6Al3kEFFP9VJhRz3DID6quisgPnTeZVr4lep9kkxdPA=
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.6.0/go.mod h1:QLvsjh0OIR0TYBeiu2bkWGTJBUNQ64st52iWj/yA93I=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+2
View File
@@ -91,6 +91,7 @@ func (c *FreedomConfig) Build() (proto.Message, error) {
}
if c.Fragment != nil {
errors.PrintDeprecatedFeatureWarning(`"fragment" in Freedom`, `"fragment" in "streamSettings.finalmask.tcp"`)
config.Fragment = new(freedom.Fragment)
switch strings.ToLower(c.Fragment.Packets) {
@@ -147,6 +148,7 @@ func (c *FreedomConfig) Build() (proto.Message, error) {
}
if c.Noises != nil {
errors.PrintDeprecatedFeatureWarning(`"noises" in Freedom`, `"noise" in "streamSettings.finalmask.udp"`)
for _, n := range c.Noises {
NConfig, err := ParseNoise(n)
if err != nil {
-4
View File
@@ -6,7 +6,6 @@ import (
"github.com/robfig/cron/v3"
"github.com/xtls/xray-core/app/geodata"
"github.com/xtls/xray-core/common/errors"
"github.com/xtls/xray-core/common/platform/filesystem"
"google.golang.org/protobuf/proto"
)
@@ -19,9 +18,6 @@ func (c *GeodataAssetConfig) Build() (*geodata.Asset, error) {
if err := validateHTTPS(c.URL); err != nil {
return nil, errors.New("invalid geodata asset url: ", c.URL).Base(err)
}
if _, err := filesystem.StatAsset(c.File); err != nil {
return nil, errors.New("invalid geodata asset file: ", c.File).Base(err)
}
return &geodata.Asset{
Url: c.URL,
File: c.File,
-23
View File
@@ -1,7 +1,6 @@
package conf_test
import (
"path/filepath"
"testing"
"github.com/xtls/xray-core/app/geodata"
@@ -9,8 +8,6 @@ import (
)
func TestGeodataConfig(t *testing.T) {
t.Setenv("xray.location.asset", filepath.Join("..", "..", "resources"))
creator := func() Buildable {
return new(GeodataConfig)
}
@@ -38,27 +35,7 @@ func TestGeodataConfig(t *testing.T) {
})
}
func TestGeodataAssetConfig(t *testing.T) {
t.Setenv("xray.location.asset", filepath.Join("..", "..", "resources"))
if _, err := (&GeodataAssetConfig{
URL: "https://example.com/geoip.dat",
File: "geoip.dat",
}).Build(); err != nil {
t.Fatal(err)
}
if _, err := (&GeodataAssetConfig{
URL: "https://example.com/geoip.dat",
File: "missing.dat",
}).Build(); err == nil {
t.Fatal("expected error")
}
}
func TestGeodataAssetConfigInvalidURL(t *testing.T) {
t.Setenv("xray.location.asset", filepath.Join("..", "..", "resources"))
for _, rawURL := range []string{
"",
"http://example.com/geoip.dat",
+24 -1
View File
@@ -997,12 +997,20 @@ type XDriveConfig struct {
EagerWindowMs uint32 `json:"eagerWindowMs"`
HoleTimeoutMs uint32 `json:"holeTimeoutMs"`
Template json.RawMessage `json:"template"`
S3 *XDriveS3Config `json:"s3"`
Xmux XmuxConfig `json:"xmux"`
// set by StreamConfig when "alpn" is ["h3"]
h3 bool
}
type XDriveS3Config struct {
Endpoint string `json:"endpoint"`
Region string `json:"region"`
Bucket string `json:"bucket"`
PathStyle *bool `json:"pathStyle"`
}
// Build implements Buildable.
func (c *XDriveConfig) Build() (proto.Message, error) {
switch c.Service {
@@ -1019,8 +1027,12 @@ func (c *XDriveConfig) Build() (proto.Message, error) {
if len(c.Template) == 0 {
return nil, errors.New(`service "template" needs a "template" object`)
}
case "s3":
if c.S3 == nil {
return nil, errors.New(`service "s3" needs an "s3" object`)
}
default:
return nil, errors.New(`unknown "service": "`, c.Service, `", use "google-drive", "template" or "local"`)
return nil, errors.New(`unknown "service": "`, c.Service, `", use "google-drive", "template", "s3" or "local"`)
}
if c.Xmux.MaxConnections.To > 0 && c.Xmux.MaxConcurrency.To > 0 {
return nil, errors.New("maxConnections cannot be specified together with maxConcurrency")
@@ -1059,5 +1071,16 @@ func (c *XDriveConfig) Build() (proto.Message, error) {
HKeepAlivePeriod: c.Xmux.HKeepAlivePeriod,
},
}
if c.Service == "s3" {
config.S3 = &xdrive.S3Config{
Endpoint: c.S3.Endpoint,
Region: c.S3.Region,
Bucket: c.S3.Bucket,
PathStyle: c.S3.PathStyle == nil || *c.S3.PathStyle,
}
if err := xdrive.ValidateS3Config(config); err != nil {
return nil, err
}
}
return config, nil
}
+1 -1
View File
@@ -324,7 +324,7 @@ func TestXDriveRejectsUnknownService(t *testing.T) {
if err := json.Unmarshal([]byte(`{"remoteFolder": "/tmp/xdrive", "service": "Dropbox"}`), config); err != nil {
t.Fatalf("Unmarshal: %v", err)
}
if _, err := config.Build(); err == nil || !strings.Contains(err.Error(), `use "google-drive", "template" or "local"`) {
if _, err := config.Build(); err == nil || !strings.Contains(err.Error(), `use "google-drive", "template", "s3" or "local"`) {
t.Fatalf("expected an unknown service error, got %v", err)
}
}
+10 -15
View File
@@ -6,9 +6,7 @@ import (
"math/big"
"net"
"runtime"
"slices"
"strconv"
"strings"
"github.com/xtls/xray-core/common/errors"
"github.com/xtls/xray-core/proxy/tun"
@@ -39,25 +37,22 @@ func (v *TunConfig) Build() (proto.Message, error) {
AutoSystemRoutingTable: v.AutoSystemRoutingTable,
AutoSystemDnsToGateway: v.AutoSystemDnsToGateway,
}
for _, leak := range v.AutoSystemWfpBlockLeak {
switch leak := strings.ToLower(leak); leak {
case "dns", "misconfigtun":
config.AutoSystemWfpBlockLeak = append(config.AutoSystemWfpBlockLeak, leak)
default:
return nil, errors.New("unknown autoSystemWfpBlockLeak value: ", leak)
}
leaks, err := tun.ExpandWfpBlockLeak(v.AutoSystemWfpBlockLeak)
if err != nil {
return nil, err
}
config.AutoSystemWfpBlockLeak = leaks
// Each option needs other settings on the system it takes effect on: the
// filters go along with the routes of autoSystemRoutingTable, "dns" lets
// DNS through the TUN only, and autoSystemDnsToGateway points the system
// DNS at the gateway.
// filters go along with the routes of autoSystemRoutingTable, keeping the
// system's DNS inside the TUN needs DNS servers on it, and
// autoSystemDnsToGateway points the system DNS at the gateway.
switch runtime.GOOS {
case "windows":
if len(config.AutoSystemWfpBlockLeak) > 0 && len(v.AutoSystemRoutingTable) == 0 {
if len(leaks) > 0 && len(v.AutoSystemRoutingTable) == 0 {
return nil, errors.New("autoSystemWfpBlockLeak needs autoSystemRoutingTable to be set")
}
if slices.Contains(config.AutoSystemWfpBlockLeak, "dns") && len(v.DNS) == 0 {
return nil, errors.New(`autoSystemWfpBlockLeak "dns" needs dns to be set`)
if tun.BlocksSystemDNS(leaks) && len(v.DNS) == 0 {
return nil, errors.New(`autoSystemWfpBlockLeak "dns", "udp/53", "tcp/53" or "windnsclient" needs dns to be set`)
}
case "linux", "freebsd":
if v.AutoSystemDnsToGateway && len(v.Gateway) == 0 {
+18 -8
View File
@@ -29,12 +29,17 @@ func TestTunConfigAutoSystem(t *testing.T) {
{
Input: `{"name": "xray0", "dns": ["1.1.1.1"], "autoSystemRoutingTable": ["0.0.0.0/0"], "autoSystemWfpBlockLeak": ["dns", "misconfigtun"]}`,
Parser: loadJSON(creator),
Output: &tun.Config{Name: "xray0", Desc: "Wintun", MTU: 1500, DNS: []string{"1.1.1.1"}, AutoSystemRoutingTable: []string{"0.0.0.0/0"}, AutoOutboundsInterface: "auto", AutoSystemWfpBlockLeak: []string{"dns", "misconfigtun"}},
Output: &tun.Config{Name: "xray0", Desc: "Wintun", MTU: 1500, DNS: []string{"1.1.1.1"}, AutoSystemRoutingTable: []string{"0.0.0.0/0"}, AutoOutboundsInterface: "auto", AutoSystemWfpBlockLeak: []string{"udp/53", "tcp/53", "windnsclient", "tcp/853", "udp/853", "misconfigtun"}},
},
{
Input: `{"name": "xray0", "dns": ["1.1.1.1"], "autoSystemRoutingTable": ["0.0.0.0/0"], "autoSystemWfpBlockLeak": ["DNS"]}`,
Parser: loadJSON(creator),
Output: &tun.Config{Name: "xray0", Desc: "Wintun", MTU: 1500, DNS: []string{"1.1.1.1"}, AutoSystemRoutingTable: []string{"0.0.0.0/0"}, AutoOutboundsInterface: "auto", AutoSystemWfpBlockLeak: []string{"dns"}},
Output: &tun.Config{Name: "xray0", Desc: "Wintun", MTU: 1500, DNS: []string{"1.1.1.1"}, AutoSystemRoutingTable: []string{"0.0.0.0/0"}, AutoOutboundsInterface: "auto", AutoSystemWfpBlockLeak: []string{"udp/53", "tcp/53", "windnsclient", "tcp/853", "udp/853"}},
},
{
Input: `{"name": "xray0", "autoSystemRoutingTable": ["0.0.0.0/0"], "autoSystemWfpBlockLeak": ["TCP/853", "udp/853", "tcp/0853"]}`,
Parser: loadJSON(creator),
Output: &tun.Config{Name: "xray0", Desc: "Wintun", MTU: 1500, AutoSystemRoutingTable: []string{"0.0.0.0/0"}, AutoOutboundsInterface: "auto", AutoSystemWfpBlockLeak: []string{"tcp/853", "udp/853"}},
},
})
}
@@ -49,6 +54,9 @@ func TestTunConfigAutoSystemNeeds(t *testing.T) {
{`{"name": "xray0", "autoSystemWfpBlockLeak": ["misconfigtun"]}`, []string{"windows"}},
{`{"name": "xray0", "autoSystemRoutingTable": ["0.0.0.0/0"], "autoSystemWfpBlockLeak": ["misconfigtun"]}`, nil},
{`{"name": "xray0", "autoSystemRoutingTable": ["0.0.0.0/0"], "autoSystemWfpBlockLeak": ["dns"]}`, []string{"windows"}},
{`{"name": "xray0", "autoSystemRoutingTable": ["0.0.0.0/0"], "autoSystemWfpBlockLeak": ["udp/53"]}`, []string{"windows"}},
{`{"name": "xray0", "autoSystemRoutingTable": ["0.0.0.0/0"], "autoSystemWfpBlockLeak": ["windnsclient"]}`, []string{"windows"}},
{`{"name": "xray0", "autoSystemRoutingTable": ["0.0.0.0/0"], "autoSystemWfpBlockLeak": ["tcp/853", "udp/853"]}`, nil},
{`{"name": "xray0", "autoSystemDnsToGateway": true}`, []string{"linux", "freebsd"}},
} {
config := new(TunConfig)
@@ -62,11 +70,13 @@ func TestTunConfigAutoSystemNeeds(t *testing.T) {
}
func TestTunConfigAutoSystemWfpBlockLeakUnknown(t *testing.T) {
config := new(TunConfig)
if err := json.Unmarshal([]byte(`{"name": "xray0", "autoSystemWfpBlockLeak": ["dns", "ip"]}`), config); err != nil {
t.Fatal(err)
}
if _, err := config.Build(); err == nil {
t.Error("an unknown autoSystemWfpBlockLeak value was accepted")
for _, value := range []string{"ip", "tcp/0", "icmp/1"} {
config := new(TunConfig)
if err := json.Unmarshal([]byte(`{"name": "xray0", "autoSystemWfpBlockLeak": ["dns", "`+value+`"]}`), config); err != nil {
t.Fatal(err)
}
if _, err := config.Build(); err == nil {
t.Errorf("autoSystemWfpBlockLeak value %q was accepted", value)
}
}
}
+6 -3
View File
@@ -237,10 +237,11 @@ type OutboundDetourConfig struct {
func requiresTransportSecurity(address *Address) bool {
if address == nil || address.Address == nil {
return false
return true
}
if address.Family().IsIP() {
return !geodata.GetPrivateIPMatcher().Match(address.IP())
ip := address.IP()
return !ip.IsPrivate() && !ip.IsLoopback() && !ip.IsLinkLocalUnicast()
}
domain := strings.TrimSuffix(strings.ToLower(address.Domain()), ".")
return !geodata.GetPrivateDomainMatcher().MatchAny(domain)
@@ -608,7 +609,6 @@ func (c *Config) Build() (*core.Config, error) {
config := &core.Config{
App: []*serial.TypedMessage{
serial.ToTypedMessage(&dispatcher.Config{}),
serial.ToTypedMessage(&proxyman.InboundConfig{}),
serial.ToTypedMessage(&proxyman.OutboundConfig{}),
},
}
@@ -718,6 +718,9 @@ func (c *Config) Build() (*core.Config, error) {
config.App = append(config.App, serial.ToTypedMessage(r))
}
// Initialize the inbound module last to ensure inbound connections are handled correctly.
config.App = append(config.App, serial.ToTypedMessage(&proxyman.InboundConfig{}))
var inbounds []InboundDetourConfig
if len(c.InboundConfigs) > 0 {
+1 -1
View File
@@ -90,7 +90,6 @@ func TestXrayConfig(t *testing.T) {
AccessLogPath: "/var/log/xray/access.log",
}),
serial.ToTypedMessage(&dispatcher.Config{}),
serial.ToTypedMessage(&proxyman.InboundConfig{}),
serial.ToTypedMessage(&proxyman.OutboundConfig{}),
serial.ToTypedMessage(&router.Config{
DomainStrategy: router.Config_AsIs,
@@ -111,6 +110,7 @@ func TestXrayConfig(t *testing.T) {
},
},
}),
serial.ToTypedMessage(&proxyman.InboundConfig{}),
},
Inbound: []*core.InboundHandlerConfig{
{
+176 -197
View File
@@ -1,149 +1,87 @@
package main
import (
"errors"
"bytes"
"context"
"flag"
"fmt"
"go/build"
"io"
"net/http"
"io/fs"
"os"
"os/exec"
"path/filepath"
"regexp"
"runtime"
"strconv"
"strings"
"github.com/bufbuild/protocompile"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/reflect/protodesc"
"google.golang.org/protobuf/reflect/protoreflect"
"google.golang.org/protobuf/types/descriptorpb"
"google.golang.org/protobuf/types/pluginpb"
)
var directory = flag.String("pwd", "", "Working directory of Xray vprotogen.")
var (
directory = flag.String("pwd", "", "Working directory of Xray vprotogen.")
mode = flag.String("mode", "generate", "Execution mode. 'generate' (default) compiles and saves changes. 'check' reports out-of-date files and exits with non-zero.")
)
// envFile returns the name of the Go environment configuration file.
// Copy from https://github.com/golang/go/blob/c4f2a9788a7be04daf931ac54382fbe2cb754938/src/cmd/go/internal/cfg/cfg.go#L150-L166
func envFile() (string, error) {
if file := os.Getenv("GOENV"); file != "" {
if file == "off" {
return "", errors.New("GOENV=off")
// topoSort sorts file descriptors so each dependency appears before files importing it.
func topoSort(files []protoreflect.FileDescriptor) []*descriptorpb.FileDescriptorProto {
fileMap := make(map[string]protoreflect.FileDescriptor)
for _, f := range files {
fileMap[f.Path()] = f
}
visited := make(map[string]bool)
var sorted []*descriptorpb.FileDescriptorProto
var visit func(fd protoreflect.FileDescriptor)
visit = func(fd protoreflect.FileDescriptor) {
if visited[fd.Path()] {
return
}
return file, nil
}
dir, err := os.UserConfigDir()
if err != nil {
return "", err
}
if dir == "" {
return "", errors.New("missing user-config dir")
}
return filepath.Join(dir, "go", "env"), nil
}
visited[fd.Path()] = true
// GetRuntimeEnv returns the value of runtime environment variable,
// that is set by running following command: `go env -w key=value`.
func GetRuntimeEnv(key string) (string, error) {
file, err := envFile()
if err != nil {
return "", err
}
if file == "" {
return "", errors.New("missing runtime env file")
}
var data []byte
var runtimeEnv string
data, readErr := os.ReadFile(file)
if readErr != nil {
return "", readErr
}
envStrings := strings.Split(string(data), "\n")
for _, envItem := range envStrings {
envItem = strings.TrimSuffix(envItem, "\r")
envKeyValue := strings.Split(envItem, "=")
if strings.EqualFold(strings.TrimSpace(envKeyValue[0]), key) {
runtimeEnv = strings.TrimSpace(envKeyValue[1])
imports := fd.Imports()
for i := 0; i < imports.Len(); i++ {
if dep, ok := fileMap[imports.Get(i).Path()]; ok {
visit(dep)
}
}
sorted = append(sorted, protodesc.ToFileDescriptorProto(fd))
}
return runtimeEnv, nil
for _, f := range files {
visit(f)
}
return sorted
}
// GetGOBIN returns GOBIN environment variable as a string. It will NOT be empty.
func GetGOBIN() string {
// The one set by user explicitly by `export GOBIN=/path` or `env GOBIN=/path command`
GOBIN := os.Getenv("GOBIN")
if GOBIN == "" {
var err error
// The one set by user by running `go env -w GOBIN=/path`
GOBIN, err = GetRuntimeEnv("GOBIN")
if err != nil {
// The default one that Golang uses
return filepath.Join(build.Default.GOPATH, "bin")
}
if GOBIN == "" {
return filepath.Join(build.Default.GOPATH, "bin")
}
return GOBIN
}
return GOBIN
}
func whichProtoc(suffix, targetedVersion string) (string, error) {
protoc := "protoc" + suffix
path, err := exec.LookPath(protoc)
func runPlugin(pluginName, pwd string, req *pluginpb.CodeGeneratorRequest) (*pluginpb.CodeGeneratorResponse, error) {
reqBytes, err := proto.Marshal(req)
if err != nil {
return "", fmt.Errorf(`
Command "%s" not found.
Make sure that %s is in your system path or current path.
Download %s v%s or later from https://github.com/protocolbuffers/protobuf/releases
`, protoc, protoc, protoc, targetedVersion)
return nil, err
}
return path, nil
}
func getProjectProtocVersion(url string) (string, error) {
resp, err := http.Get(url)
if err != nil {
return "", errors.New("can not get the version of protobuf used in xray project")
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return "", errors.New("can not read from body")
}
versionRegexp := regexp.MustCompile(`\/\/\s*protoc\s*v\d+\.(\d+\.\d+)`)
matched := versionRegexp.FindStringSubmatch(string(body))
return matched[1], nil
}
cmd := exec.Command("go", "tool", pluginName)
cmd.Dir = pwd
cmd.Stdin = bytes.NewReader(reqBytes)
cmd.Env = os.Environ()
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
func getInstalledProtocVersion(protocPath string) (string, error) {
cmd := exec.Command(protocPath, "--version")
cmd.Env = append(cmd.Env, os.Environ()...)
output, cmdErr := cmd.CombinedOutput()
if cmdErr != nil {
return "", cmdErr
if err := cmd.Run(); err != nil {
return nil, fmt.Errorf("%v: %s", err, stderr.String())
}
versionRegexp := regexp.MustCompile(`protoc\s*(\d+\.\d+)`)
matched := versionRegexp.FindStringSubmatch(string(output))
return matched[1], nil
}
func parseVersion(s string, width int) int64 {
strList := strings.Split(s, ".")
format := fmt.Sprintf("%%s%%0%ds", width)
v := ""
for _, value := range strList {
v = fmt.Sprintf(format, v, value)
var resp pluginpb.CodeGeneratorResponse
if err := proto.Unmarshal(stdout.Bytes(), &resp); err != nil {
return nil, fmt.Errorf("failed to unmarshal response: %w", err)
}
var result int64
var err error
if result, err = strconv.ParseInt(v, 10, 64); err != nil {
return 0
if resp.GetError() != "" {
return nil, fmt.Errorf("%s", resp.GetError())
}
return result
}
func needToUpdate(targetedVersion, installedVersion string) bool {
vt := parseVersion(targetedVersion, 4)
vi := parseVersion(installedVersion, 4)
return vt > vi
return &resp, nil
}
func main() {
@@ -154,106 +92,147 @@ func main() {
flag.Parse()
if !filepath.IsAbs(*directory) {
pwd, wdErr := os.Getwd()
if wdErr != nil {
pwd, err := os.Getwd()
if err != nil {
fmt.Println("Can not get current working directory.")
os.Exit(1)
}
*directory = filepath.Join(pwd, *directory)
}
pwd := *directory
GOBIN := GetGOBIN()
binPath := os.Getenv("PATH")
pathSlice := []string{pwd, GOBIN, binPath}
binPath = strings.Join(pathSlice, string(os.PathListSeparator))
os.Setenv("PATH", binPath)
suffix := ""
if runtime.GOOS == "windows" {
suffix = ".exe"
}
/*
targetedVersion, err := getProjectProtocVersion("https://raw.githubusercontent.com/XTLS/Xray-core/HEAD/core/config.pb.go")
if err != nil {
fmt.Println(err)
os.Exit(1)
}
*/
targetedVersion := ""
protoc, err := whichProtoc(suffix, targetedVersion)
if err != nil {
fmt.Println(err)
isCheck := false
switch *mode {
case "generate", "":
isCheck = false
case "check":
isCheck = true
default:
fmt.Fprintf(os.Stderr, "Unrecognized 'mode': %s. Use 'generate' or 'check'.\n", *mode)
os.Exit(1)
}
installedVersion, err := getInstalledProtocVersion(protoc)
if err != nil {
fmt.Println(err)
os.Exit(1)
if isCheck {
fmt.Println("Checking for out-of-date Protobuf generated files...")
}
if needToUpdate(targetedVersion, installedVersion) {
fmt.Printf(`
You are using an old protobuf version, please update to v%s or later.
Download it from https://github.com/protocolbuffers/protobuf/releases
* Protobuf version used in xray project: v%s
* Protobuf version you have installed: v%s
`, targetedVersion, targetedVersion, installedVersion)
os.Exit(1)
}
protoFilesMap := make(map[string][]string)
walkErr := filepath.Walk(pwd, func(path string, info os.FileInfo, err error) error {
if err != nil {
fmt.Println(err)
var protoFiles []string
var pbCount int
err := filepath.WalkDir(pwd, func(path string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return err
}
if info.IsDir() {
return nil
if strings.HasSuffix(d.Name(), ".proto") {
rel, _ := filepath.Rel(pwd, path)
protoFiles = append(protoFiles, filepath.ToSlash(rel))
}
dir := filepath.Dir(path)
filename := filepath.Base(path)
if strings.HasSuffix(filename, ".proto") {
path = path[len(pwd)+1:]
protoFilesMap[dir] = append(protoFilesMap[dir], path)
if strings.HasSuffix(d.Name(), ".pb.go") {
pbCount++
}
return nil
})
if walkErr != nil {
fmt.Println(walkErr)
if err != nil {
fmt.Println("Walk error:", err)
os.Exit(1)
}
if len(protoFiles) == 0 {
if isCheck && pbCount > 0 {
fmt.Fprintf(os.Stderr, "Found %d .pb.go files without any .proto files. Remove these orphaned files.\n", pbCount)
os.Exit(1)
}
return
}
compiler := protocompile.Compiler{
Resolver: protocompile.WithStandardImports(&protocompile.SourceResolver{
ImportPaths: []string{pwd},
}),
SourceInfoMode: protocompile.SourceInfoStandard,
}
linkedFiles, err := compiler.Compile(context.Background(), protoFiles...)
if err != nil {
fmt.Fprintf(os.Stderr, "Protobuf compilation failed:\n%v\n", err)
os.Exit(1)
}
for _, files := range protoFilesMap {
for _, relProtoFile := range files {
args := []string{
"--go_out", pwd,
"--go_opt", "paths=source_relative",
"--go-grpc_out", pwd,
"--go-grpc_opt", "paths=source_relative",
"--plugin", "protoc-gen-go=" + filepath.Join(GOBIN, "protoc-gen-go"+suffix),
"--plugin", "protoc-gen-go-grpc=" + filepath.Join(GOBIN, "protoc-gen-go-grpc"+suffix),
var fdList []protoreflect.FileDescriptor
for _, f := range linkedFiles {
fdList = append(fdList, f)
}
req := &pluginpb.CodeGeneratorRequest{
FileToGenerate: protoFiles,
Parameter: proto.String("paths=source_relative"),
ProtoFile: topoSort(fdList),
// TODO: replace with unknown in the future
CompilerVersion: &pluginpb.Version{
Major: proto.Int32(6),
Minor: proto.Int32(33),
Patch: proto.Int32(5),
},
}
var diffCount int
generatedFiles := make(map[string]struct{})
for _, plugin := range []string{"protoc-gen-go", "protoc-gen-go-grpc"} {
resp, err := runPlugin(plugin, pwd, req)
if err != nil {
fmt.Fprintf(os.Stderr, "%s failed: %v\n", plugin, err)
os.Exit(1)
}
for _, f := range resp.File {
targetPath := filepath.Join(pwd, filepath.FromSlash(f.GetName()))
generatedFiles[targetPath] = struct{}{}
content := []byte(f.GetContent())
hasDiff := false
existing, err := os.ReadFile(targetPath)
if err != nil {
hasDiff = true
} else {
oldNorm := bytes.ReplaceAll(existing, []byte("\r\n"), []byte("\n"))
newNorm := bytes.ReplaceAll(content, []byte("\r\n"), []byte("\n"))
if !bytes.Equal(oldNorm, newNorm) {
hasDiff = true
}
}
args = append(args, relProtoFile)
cmd := exec.Command(protoc, args...)
cmd.Env = append(cmd.Env, os.Environ()...)
cmd.Dir = pwd
output, cmdErr := cmd.CombinedOutput()
if len(output) > 0 {
fmt.Println(string(output))
}
if cmdErr != nil {
fmt.Println(cmdErr)
os.Exit(1)
if hasDiff {
diffCount++
fmt.Println(targetPath)
if isCheck {
continue
}
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
fmt.Fprintf(os.Stderr, "Error creating directory: %v\n", err)
os.Exit(1)
}
mode := os.FileMode(0o644)
if stat, err := os.Stat(targetPath); err == nil {
mode = stat.Mode().Perm()
}
if err := os.WriteFile(targetPath, content, mode); err != nil {
fmt.Fprintf(os.Stderr, "Error writing %s: %v\n", targetPath, err)
os.Exit(1)
}
}
}
}
if isCheck {
if pbCount != len(generatedFiles) {
fmt.Printf("Protobuf file count mismatch: found %d, expected %d. Remove orphaned files and regenerate.\n", pbCount, len(generatedFiles))
diffCount++
}
if diffCount > 0 {
fmt.Println("Protobuf generated file problem(s) found.")
fmt.Println("Please run 'go run ./infra/vprotogen' to regenerate them.")
os.Exit(1)
}
fmt.Println("All Protobuf generated file check has been passed.")
}
}
+2
View File
@@ -92,6 +92,8 @@ func executeAddRules(cmd *base.Command, args []string) {
if err != nil {
base.Fatalf("failed to perform AddRule: %s", err)
}
// if multiple files are specified, use append to merge them instead of overriding previous one
shouldAppend = true
showJSONResponse(resp)
}
}
+4 -11
View File
@@ -3,6 +3,7 @@ package lua
import (
"os"
"path/filepath"
"strings"
xlua "github.com/xtls/xray-core/common/lua"
"github.com/xtls/xray-core/main/commands/base"
@@ -32,14 +33,13 @@ func executeSign(cmd *base.Command, args []string) {
cmd.Usage()
}
if !strings.EqualFold(filepath.Ext(args[0]), ".lua") {
base.Fatalf("Lua file extension must be .lua")
}
info, err := os.Stat(args[0])
if err != nil {
base.Fatalf("failed to inspect Lua file: %s", err)
}
if !info.Mode().IsRegular() {
base.Fatalf("Lua input is not a regular file: %s", args[0])
}
code, err := os.ReadFile(args[0])
if err != nil {
base.Fatalf("failed to read Lua file: %s", err)
@@ -67,14 +67,7 @@ func executeSign(cmd *base.Command, args []string) {
func writeSignedFile(path string, code []byte, mode os.FileMode) error {
if info, err := os.Stat(path); err == nil {
if !info.Mode().IsRegular() {
return &os.PathError{Op: "write", Path: path, Err: os.ErrInvalid}
}
mode = info.Mode().Perm()
path, err = filepath.EvalSymlinks(path)
if err != nil {
return err
}
} else if !os.IsNotExist(err) {
return err
}
+2 -10
View File
@@ -58,18 +58,10 @@ func readConfigFile(file *os.File) ([]byte, error) {
return nil, err
}
if info.Mode().IsRegular() {
path, err := filepath.Abs(file.Name())
if err != nil {
return nil, err
}
path, err = filepath.EvalSymlinks(path)
if err != nil {
return nil, err
}
switch strings.ToLower(filepath.Ext(path)) {
switch strings.ToLower(filepath.Ext(file.Name())) {
case ".jsonc", ".json", ".yml", ".yaml", ".toml", ".pb", ".protobuf":
default:
return nil, errors.New("unsupported config file extension: ", path)
return nil, errors.New("unsupported config file extension: ", file.Name())
}
}
return io.ReadAll(file)
+7
View File
@@ -381,6 +381,13 @@ func (h *Handler) handleIPQuery(id uint16, qType dnsmessage.Type, domain string,
errors.LogInfoInner(context.Background(), err, "pack message")
b.Release()
timer.SetTimeout(0)
return
}
if len(msgBytes) > buf.Size {
b.Release()
timer.SetTimeout(0)
errors.LogError(context.Background(), "DNS response message too large")
return
}
b.Resize(0, int32(len(msgBytes)))
+22 -22
View File
@@ -23,14 +23,14 @@ const (
)
type Config struct {
state protoimpl.MessageState `protogen:"open.v1"`
RewriteAddress *net.IPOrDomain `protobuf:"bytes,1,opt,name=rewrite_address,json=rewriteAddress,proto3" json:"rewrite_address,omitempty"`
RewritePort uint32 `protobuf:"varint,2,opt,name=rewrite_port,json=rewritePort,proto3" json:"rewrite_port,omitempty"`
PortMap map[string]string `protobuf:"bytes,3,rep,name=port_map,json=portMap,proto3" json:"port_map,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
state protoimpl.MessageState `protogen:"open.v1"`
// List of networks that the Dokodemo accepts.
AllowedNetworks []net.Network `protobuf:"varint,7,rep,packed,name=allowed_networks,json=allowedNetworks,proto3,enum=xray.common.net.Network" json:"allowed_networks,omitempty"`
FollowRedirect bool `protobuf:"varint,5,opt,name=follow_redirect,json=followRedirect,proto3" json:"follow_redirect,omitempty"`
UserLevel uint32 `protobuf:"varint,6,opt,name=user_level,json=userLevel,proto3" json:"user_level,omitempty"`
AllowedNetworks []net.Network `protobuf:"varint,7,rep,packed,name=allowed_networks,json=allowedNetworks,proto3,enum=xray.common.net.Network" json:"allowed_networks,omitempty"`
RewriteAddress *net.IPOrDomain `protobuf:"bytes,1,opt,name=rewrite_address,json=rewriteAddress,proto3" json:"rewrite_address,omitempty"`
RewritePort uint32 `protobuf:"varint,2,opt,name=rewrite_port,json=rewritePort,proto3" json:"rewrite_port,omitempty"`
PortMap map[string]string `protobuf:"bytes,3,rep,name=port_map,json=portMap,proto3" json:"port_map,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
FollowRedirect bool `protobuf:"varint,5,opt,name=follow_redirect,json=followRedirect,proto3" json:"follow_redirect,omitempty"`
UserLevel uint32 `protobuf:"varint,6,opt,name=user_level,json=userLevel,proto3" json:"user_level,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -65,6 +65,13 @@ func (*Config) Descriptor() ([]byte, []int) {
return file_proxy_dokodemo_config_proto_rawDescGZIP(), []int{0}
}
func (x *Config) GetAllowedNetworks() []net.Network {
if x != nil {
return x.AllowedNetworks
}
return nil
}
func (x *Config) GetRewriteAddress() *net.IPOrDomain {
if x != nil {
return x.RewriteAddress
@@ -86,13 +93,6 @@ func (x *Config) GetPortMap() map[string]string {
return nil
}
func (x *Config) GetAllowedNetworks() []net.Network {
if x != nil {
return x.AllowedNetworks
}
return nil
}
func (x *Config) GetFollowRedirect() bool {
if x != nil {
return x.FollowRedirect
@@ -112,11 +112,11 @@ var File_proxy_dokodemo_config_proto protoreflect.FileDescriptor
const file_proxy_dokodemo_config_proto_rawDesc = "" +
"\n" +
"\x1bproxy/dokodemo/config.proto\x12\x13xray.proxy.dokodemo\x1a\x18common/net/address.proto\x1a\x18common/net/network.proto\"\xff\x02\n" +
"\x06Config\x12D\n" +
"\x06Config\x12C\n" +
"\x10allowed_networks\x18\a \x03(\x0e2\x18.xray.common.net.NetworkR\x0fallowedNetworks\x12D\n" +
"\x0frewrite_address\x18\x01 \x01(\v2\x1b.xray.common.net.IPOrDomainR\x0erewriteAddress\x12!\n" +
"\frewrite_port\x18\x02 \x01(\rR\vrewritePort\x12C\n" +
"\bport_map\x18\x03 \x03(\v2(.xray.proxy.dokodemo.Config.PortMapEntryR\aportMap\x12C\n" +
"\x10allowed_networks\x18\a \x03(\x0e2\x18.xray.common.net.NetworkR\x0fallowedNetworks\x12'\n" +
"\bport_map\x18\x03 \x03(\v2(.xray.proxy.dokodemo.Config.PortMapEntryR\aportMap\x12'\n" +
"\x0ffollow_redirect\x18\x05 \x01(\bR\x0efollowRedirect\x12\x1d\n" +
"\n" +
"user_level\x18\x06 \x01(\rR\tuserLevel\x1a:\n" +
@@ -141,13 +141,13 @@ var file_proxy_dokodemo_config_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
var file_proxy_dokodemo_config_proto_goTypes = []any{
(*Config)(nil), // 0: xray.proxy.dokodemo.Config
nil, // 1: xray.proxy.dokodemo.Config.PortMapEntry
(*net.IPOrDomain)(nil), // 2: xray.common.net.IPOrDomain
(net.Network)(0), // 3: xray.common.net.Network
(net.Network)(0), // 2: xray.common.net.Network
(*net.IPOrDomain)(nil), // 3: xray.common.net.IPOrDomain
}
var file_proxy_dokodemo_config_proto_depIdxs = []int32{
2, // 0: xray.proxy.dokodemo.Config.rewrite_address:type_name -> xray.common.net.IPOrDomain
1, // 1: xray.proxy.dokodemo.Config.port_map:type_name -> xray.proxy.dokodemo.Config.PortMapEntry
3, // 2: xray.proxy.dokodemo.Config.allowed_networks:type_name -> xray.common.net.Network
2, // 0: xray.proxy.dokodemo.Config.allowed_networks:type_name -> xray.common.net.Network
3, // 1: xray.proxy.dokodemo.Config.rewrite_address:type_name -> xray.common.net.IPOrDomain
1, // 2: xray.proxy.dokodemo.Config.port_map:type_name -> xray.proxy.dokodemo.Config.PortMapEntry
3, // [3:3] is the sub-list for method output_type
3, // [3:3] is the sub-list for method input_type
3, // [3:3] is the sub-list for extension type_name
+10 -5
View File
@@ -229,14 +229,19 @@ func (h *Handler) blockDelay(rule *FinalRule) time.Duration {
func (h *Handler) blackhole(ctx context.Context, input buf.Reader, output buf.Writer, rule *FinalRule, dest *net.Destination) error {
delay := h.blockDelay(rule)
errors.LogInfo(ctx, "blocked target: ", *dest, ", blackholing connection for ", delay)
timer := time.AfterFunc(delay, func() {
defer common.Close(output)
done := make(chan struct{})
go func() { // a DispatchLink inbound's reader ignores Interrupt, so don't wait for this copy
_ = buf.Copy(input, buf.Discard)
close(done)
}()
select {
case <-done:
case <-time.After(delay):
common.Interrupt(input)
common.Interrupt(output)
errors.LogInfo(ctx, "closed blackholed connection to blocked target: ", *dest)
})
defer timer.Stop()
defer common.Close(output)
_ = buf.Copy(input, buf.Discard)
}
return nil
}
+15 -4
View File
@@ -23,6 +23,10 @@ For more advanced routing policies or rules, OS level configuration can still ma
This keeps complex system level routing and rules in a single place of responsibility - the OS itself. \
Examples of how to achieve this on a simple Linux system (Ubuntu with systemd-networkd) can be found at the end of this README.
Packets are handled by Xray-core's own lightweight TCP/IP stack, which replaced gVisor (XTLS/Xray-core#7130).
With `autoSystemRoutingTable`, Xray's own connections are kept out of the TUN through `autoOutboundsInterface` (then `"auto"` by default): each of their sockets is bound to another interface, the one named there, or with `"auto"` the one of the default route (Android needs the name). Linux and Android bind with `SO_BINDTODEVICE`, Windows with `IP_UNICAST_IF` and `IPV6_UNICAST_IF`, macOS and iOS with `IP_BOUND_IF` and `IPV6_BOUND_IF`, and FreeBSD switches them to another routing table (see [FreeBSD SUPPORT](#freebsd-support)). Connections to loopback addresses, TCP and UDP, stay unbound. If a socket cannot be bound, the connection fails rather than leave unbound, where it could loop back into the TUN (XTLS/Xray-core#7173).
### SYSTEM DNS ON LINUX AND FREEBSD (`autoSystemDnsToGateway`)
On Linux and FreeBSD, setting `autoSystemDnsToGateway` to `true` lets the inbound point the system resolver at the tun interface, so name lookups resolve through Xray instead of going out over the physical link. It is off by default, and it exists only on those two systems. This section describes Linux; the FreeBSD differences follow in [SYSTEM DNS ON FREEBSD](#system-dns-on-freebsd).
@@ -215,12 +219,16 @@ All in all systemd-networkd and its derivatives (like netplan or NetworkManager)
Windows version of the same functionality is implemented through Wintun library. \
To make it start, wintun.dll specific for your Windows/arch must be present next to Xray.exe binary.
After the start network adapter with the name you chose in the config will be created in the system, and exist while Xray is running.
After the start network adapter with the name you chose in the config will be created in the system, and exist while Xray is running. A Wintun adapter that already has that name is reused.
With IPv6 disabled system-wide (the `DisabledComponents` registry value), the adapter has no IPv6, and the TUN starts with IPv4 only, unless `gateway` or `autoSystemRoutingTable` has IPv6 entries, which then keep it from starting.
When `dns` is set, those servers are applied to the adapter. Windows is kept from registering the TUN's addresses in DNS, and its DNS cache is flushed when the TUN starts and stops.
With `autoSystemWfpBlockLeak`, which needs `autoSystemRoutingTable` (the config is rejected otherwise), Xray also adds Windows Filtering Platform filters that keep two kinds of traffic of every program but Xray itself from leaving outside the TUN, each chosen by a value in the list, e.g. `"autoSystemWfpBlockLeak": ["dns", "misconfigtun"]`:
- `"dns"` (needs `dns`, the config is rejected otherwise): DNS (port 53) only goes through the TUN. Windows keeps sending name queries to the DNS servers of the other interfaces as well, out through those interfaces whatever the routes say, and other programs reach a resolver on the local network (e.g. `192.168.1.1` handed out by DHCP) through its more specific LAN route instead of the TUN. On Windows 11 and Server 2022 and later, where those queries may also go over HTTPS or TLS, Windows' DNS Client service cannot connect outside the TUN at all, except for name resolution on the local network (LLMNR, mDNS). The `dns` servers therefore have to lie within `gateway` or `autoSystemRoutingTable` (a warning is logged otherwise), and DNS servers that should be reached directly belong in Xray's own `dns` settings.
With `autoSystemWfpBlockLeak`, which needs `autoSystemRoutingTable` (the config is rejected otherwise), Xray also adds Windows Filtering Platform filters that keep traffic of every program but Xray itself from leaving outside the TUN, each kind chosen by a value in the list, e.g. `"autoSystemWfpBlockLeak": ["dns", "misconfigtun"]`:
- `"tcp/<port>"`, `"udp/<port>"`: connections to that TCP or UDP port only go through the TUN, e.g. `"tcp/853"`.
- `"windnsclient"`: Windows' DNS Client service cannot connect outside the TUN at all, except for name resolution on the local network (LLMNR, mDNS). On Windows 11 and Server 2022 and later, it may send name queries over HTTPS or TLS, to any port. Earlier versions only query port 53, so there it does nothing.
- `"dns"`: stands for `"udp/53"`, `"tcp/53"`, `"windnsclient"`, `"tcp/853"` (DNS over TLS) and `"udp/853"` (DNS over QUIC). Windows keeps sending name queries to the DNS servers of the other interfaces as well, out through those interfaces whatever the routes say, and other programs reach a resolver on the local network (e.g. `192.168.1.1` handed out by DHCP) through its more specific LAN route instead of the TUN. Keeping the system's DNS inside the TUN like this, with `"dns"`, `"udp/53"`, `"tcp/53"` or `"windnsclient"`, needs `dns` (the config is rejected otherwise). The `dns` servers then have to lie within `gateway` or `autoSystemRoutingTable` (a warning is logged otherwise), and DNS servers that should be reached directly belong in Xray's own `dns` settings.
- `"misconfigtun"`: an IP version without routes in `autoSystemRoutingTable`, IPv4 or IPv6, is blocked entirely, in both directions, as it would bypass the TUN. Only loopback and what Windows itself needs on the local link (DHCP, and for IPv6 neighbor and multicast listener discovery) remain allowed. An address of that version in `gateway` is not needed: without one, Windows gives the TUN link-local addresses itself, an IPv6 one at once and an IPv4 one from `169.254.0.0/16` after some seconds (until then, IPv4 routed to the TUN is unreachable), and what is routed to the TUN goes through it with those.
With the filters in place, Xray's own connections out also get past Windows Firewall's block rules (other firewalls may still block them), while connections to Xray's inbounds stay subject to them.
@@ -233,6 +241,8 @@ If the filters cannot be added, Xray does not start. They are removed when Xray
`autoOutboundsInterface` (the default with `autoSystemRoutingTable`) keeps Xray's own connections out of the TUN by binding them to another interface, which Windows only honors while that interface has weak host send and forwarding off for the IP versions routed to the TUN. Otherwise, Windows sends them into the TUN, from that interface's address, and they stall. While the TUN runs, Xray therefore turns weak host send off on that interface, and on again when it stops or another interface takes over. Forwarding cannot be turned off this way, as Mobile Hotspot and Internet Connection Sharing need it, so a warning is logged while it is on. Having the hotspot share the TUN instead of that interface (Settings, Mobile hotspot, Share my internet connection from) moves forwarding to the TUN, where it does no harm, and sends the hotspot's devices through Xray as well.
An outbound interface with only IPv4 or only IPv6 works as well: a UDP socket opened for both IP versions, which such an interface cannot be bound to, is opened again for the destination's version only, and IPv6 TCP is bound for IPv6 alone, so connections of the missing version fail rather than leave through another interface (XTLS/Xray-core#7214).
You can give the adapter ip address manually, you can live Windows to give it autogenerated ip address (which take few seconds), it doesn't matter, the traffic going _through_ the interface will be forwarded into the app for proxying. \
Minimal configuration that will work for local machine is routing passing the traffic on-link through the interface.
You will need the interface id for that, unfortunately it is going to change with every Xray start due to implementation ambiguity between Xray and wintun driver.
@@ -307,6 +317,7 @@ sudo route add -inet6 -host 2606:4700:4700::1111 -iface utun10
sudo route add -inet6 -host 2606:4700:4700::1001 -iface utun10
```
Alternatively, configure `autoSystemRoutingTable` and Xray will add and remove those system routes while it is running.
Xray's own connections are then kept out of the TUN through `autoOutboundsInterface`: an IPv4 socket is bound with `IP_BOUND_IF`, an IPv6 or dual-stack one with `IPV6_BOUND_IF`, which covers the IPv4 traffic of a dual-stack socket as well, since macOS rejects `IP_BOUND_IF` on IPv6 sockets (XTLS/Xray-core#7230).
Important to remember that everything written above about Linux routing concept, also apply to Mac OS X. If you simply route default route through utun interface, that will result network loop and immediate network failure.
## ANDROID SUPPORT
@@ -338,7 +349,7 @@ let tunFd = ((0 ... 1024).first { (_ fd: Int32) -> Bool in var len = socklen_t(b
}!
```
Set the environment variable `xray.tun.fd` (or `XRAY_TUN_FD`) to the fd number before starting Xray. This can be done from Swift/Objective-C or by exposing a Go function via gomobile bindings.
Set the environment variable `xray.tun.fd` (or `XRAY_TUN_FD`) to the fd number before starting Xray. This can be done from Swift/Objective-C or by exposing a Go function via gomobile bindings. Xray works on a duplicate of the fd and closes only that, so the fd itself stays NetworkExtension's (XTLS/Xray-core#6883).
Build using gomobile for iOS framework integration:
```
+2 -3
View File
@@ -7,12 +7,11 @@
package tun
import (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
unsafe "unsafe"
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
)
const (
+4 -7
View File
@@ -558,19 +558,16 @@ func ioctlPtr(fd int, req uint, arg unsafe.Pointer) error {
}
func setinterface(network, address string, fd uintptr, iface *net.Interface) error {
var err1, err2 error
switch network {
case "tcp6", "udp6", "ip6":
err1 = unix.SetsockoptInt(int(fd), unix.IPPROTO_IPV6, unix.IPV6_BOUND_IF, iface.Index)
fallthrough
// IPV6_BOUND_IF binds the IPv4 traffic of a dual-stack socket as
// well, and an IPv6 socket rejects IP_BOUND_IF with EINVAL.
return unix.SetsockoptInt(int(fd), unix.IPPROTO_IPV6, unix.IPV6_BOUND_IF, iface.Index)
case "tcp4", "udp4", "ip4":
err2 = unix.SetsockoptInt(int(fd), unix.IPPROTO_IP, unix.IP_BOUND_IF, iface.Index)
return unix.SetsockoptInt(int(fd), unix.IPPROTO_IP, unix.IP_BOUND_IF, iface.Index)
default:
panic(network + " " + address)
}
return errors.Join(err1, err2)
}
func findOutboundInterface(tunIndex int, fixedName string) (*net.Interface, error) {
+40
View File
@@ -3,7 +3,10 @@
package tun
import (
"context"
"net"
"sync"
"syscall"
"testing"
"time"
@@ -231,3 +234,40 @@ func TestWaitKqueueReportsPersistentFailure(t *testing.T) {
}
}
}
// TestSetinterfaceSocketFamilies binds each kind of socket Xray's dialers
// open to lo0, including the dual-stack one Go opens for UDP.
func TestSetinterfaceSocketFamilies(t *testing.T) {
lo, err := net.InterfaceByName("lo0")
if err != nil {
t.Skip(err)
}
lc := net.ListenConfig{Control: func(network, address string, c syscall.RawConn) error {
var err error
if controlErr := c.Control(func(fd uintptr) {
err = setinterface(network, address, fd, lo)
}); controlErr != nil {
return controlErr
}
return err
}}
for _, s := range []struct{ network, address string }{
{"udp", "0.0.0.0:0"}, // dual-stack
{"udp4", "127.0.0.1:0"},
{"udp6", "[::1]:0"},
{"tcp4", "127.0.0.1:0"},
{"tcp6", "[::1]:0"},
} {
var closer interface{ Close() error }
if s.network[:3] == "udp" {
closer, err = lc.ListenPacket(context.Background(), s.network, s.address)
} else {
closer, err = lc.Listen(context.Background(), s.network, s.address)
}
if err != nil {
t.Errorf("%s %s: %v", s.network, s.address, err)
continue
}
closer.Close()
}
}
+49 -24
View File
@@ -247,30 +247,31 @@ startOver:
}
// With autoSystemWfpBlockLeak, once the system routes lead to the TUN,
// keep DNS ("dns", if dns is set), and an IP version no route of which
// leads to the TUN ("misconfigtun"), from leaving through the other
// interfaces. Addresses do not matter: without one of a version in
// gateway, Windows gives the TUN a link-local one.
leaks := t.options.AutoSystemWfpBlockLeak
blockDNS := slices.Contains(leaks, "dns") && len(dns) > 0
blockIPv4 := slices.Contains(leaks, "misconfigtun") && !route4
blockIPv6 := slices.Contains(leaks, "misconfigtun") && !route6
if (route4 || route6) && (blockDNS || blockIPv4 || blockIPv6) {
if t.wfp, err = blockLeaks(t.luid, blockDNS, blockIPv4, blockIPv6); err != nil {
var blocked []string
for _, b := range []struct {
on bool
what string
}{{blockDNS, "DNS"}, {blockIPv4, "IPv4"}, {blockIPv6, "IPv6"}} {
if b.on {
blocked = append(blocked, b.what)
}
}
// Rather no TUN than a leaking one.
return errors.New("unable to block ", strings.Join(blocked, " and "), " outside the TUN (remove autoSystemWfpBlockLeak to run without)").Base(err)
// keep connections to the ports given ("dns" stands for those of DNS),
// Windows' DNS Client service, and an IP version no route of which leads
// to the TUN ("misconfigtun"), from leaving through the other interfaces.
// Addresses do not matter: without one of a version in gateway, Windows
// gives the TUN a link-local one.
leaks, err := ExpandWfpBlockLeak(t.options.AutoSystemWfpBlockLeak)
if err != nil {
return err
}
block := newLeakFilters(leaks, route4, route6)
if route4 || route6 {
if slices.Contains(leaks, "windnsclient") && !block.dnsClient {
errors.LogInfo(context.Background(), "[tun] windnsclient skipped: before Windows 11 and Server 2022, the DNS Client service only queries port 53")
}
errors.LogInfo(context.Background(), "[tun] outside the TUN, blocked DNS: ", blockDNS, ", blocked IPv4: ", blockIPv4, ", blocked IPv6: ", blockIPv6)
if blockDNS {
if block.systemDNS() && len(dns) == 0 {
return errors.New("keeping the system's DNS inside the TUN needs dns to be set")
}
}
if blocked := block.String(); (route4 || route6) && blocked != "" {
if t.wfp, err = blockLeaks(t.luid, block); err != nil {
// Rather no TUN than a leaking one.
return errors.New("unable to block ", blocked, " outside the TUN (remove autoSystemWfpBlockLeak to run without)").Base(err)
}
errors.LogInfo(context.Background(), "[tun] blocked outside the TUN: ", blocked)
if block.systemDNS() {
covered := slices.Clone(addresses)
for _, route := range routesData {
covered = append(covered, route.Destination)
@@ -401,6 +402,15 @@ func resolveOnOwn() *savedResolver {
if internet.IsSkippedDNSServer(address) {
return nil, errors.New("skipped DNS server ", address)
}
// A socket of the server's IP version only needs that version bound
// to the interface, which a dual-stack one would need both of.
if ap, err := netip.ParseAddrPort(address); err == nil && (network == "udp" || network == "tcp") {
if ap.Addr().Unmap().Is4() {
network += "4"
} else {
network += "6"
}
}
return dialer.DialContext(ctx, network, address)
}
net.DefaultResolver.PreferGo = true
@@ -558,10 +568,22 @@ func setinterface(network, address string, fd uintptr, iface *net.Interface) err
switch network {
case "tcp6", "udp6", "ip6":
dualStack := false
if network == "udp6" {
ipv6Only, err := windows.GetsockoptInt(windows.Handle(fd), windows.IPPROTO_IPV6, windows.IPV6_V6ONLY)
if err != nil {
return internet.NewBindError(err)
}
dualStack = ipv6Only == 0
}
err1 = windows.SetsockoptInt(windows.Handle(fd), windows.IPPROTO_IPV6, IPV6_UNICAST_IF, iface.Index)
if network == "udp6" {
err2 = windows.SetsockoptInt(windows.Handle(fd), windows.IPPROTO_IPV6, windows.IPV6_MULTICAST_IF, iface.Index)
}
// Only dual-stack UDP can send to both IPv4 and IPv6 destinations.
if !dualStack {
break
}
fallthrough
case "tcp4", "udp4", "ip4":
err3 = windows.SetsockoptInt(windows.Handle(fd), windows.IPPROTO_IP, IP_UNICAST_IF, *(*int)(unsafe.Pointer(&index[0])))
@@ -572,7 +594,10 @@ func setinterface(network, address string, fd uintptr, iface *net.Interface) err
panic(network + " " + address)
}
return errors.Combine(err1, err2, err3, err4)
if err := errors.Combine(err1, err2, err3, err4); err != nil {
return internet.NewBindError(err)
}
return nil
}
func findOutboundInterface(tunIndex int, fixedName string) (*net.Interface, error) {
+105 -40
View File
@@ -7,6 +7,7 @@ import (
"os"
"runtime"
"slices"
"strings"
"unsafe"
"github.com/xtls/xray-core/common/errors"
@@ -170,19 +171,79 @@ func condition(field *windows.GUID, typ uint32, value uintptr) fwpmFilterConditi
}
}
// leakFilters is what blockLeaks keeps from leaving outside the TUN.
type leakFilters struct {
ports []leakPort // connections to these ports
dnsClient bool // Windows' DNS Client service
ipv4, ipv6 bool // an IP version altogether
}
// newLeakFilters turns expanded autoSystemWfpBlockLeak values into the filters
// for a TUN that the routes of route4 (IPv4) and route6 (IPv6) lead to.
func newLeakFilters(leaks []string, route4, route6 bool) leakFilters {
var block leakFilters
for _, leak := range leaks {
switch leak {
case "windnsclient":
block.dnsClient = dnsClientUsesAnyPort()
case "misconfigtun":
block.ipv4, block.ipv6 = !route4, !route6
default:
if port, ok := parseLeakPort(leak); ok {
block.ports = append(block.ports, port)
}
}
}
return block
}
// String lists what block keeps inside the TUN, or is empty for nothing.
func (block leakFilters) String() string {
var blocked []string
for _, port := range block.ports {
blocked = append(blocked, port.String())
}
for _, b := range []struct {
on bool
what string
}{{block.dnsClient, "windnsclient"}, {block.ipv4, "IPv4"}, {block.ipv6, "IPv6"}} {
if b.on {
blocked = append(blocked, b.what)
}
}
return strings.Join(blocked, ", ")
}
// systemDNS tells whether block keeps the system's DNS inside the TUN.
func (block leakFilters) systemDNS() bool {
return block.dnsClient || slices.ContainsFunc(block.ports, func(port leakPort) bool { return port.port == 53 })
}
// dnsClientUsesAnyPort tells whether Windows' DNS Client service may send its
// queries over HTTPS or TLS, and so to any port, which it does since Windows 11
// and Server 2022 (build 20348). Earlier versions only query port 53, and may
// run the service in one process with others, which its filters would catch
// as well.
func dnsClientUsesAnyPort() bool {
_, _, build := windows.RtlGetNtVersionNumbers()
return build >= 20348
}
// blockLeaks keeps traffic from leaving through interfaces other than tun,
// for every program but Xray itself, whose outbounds (DNS included) use the
// other interfaces on purpose:
//
// - dns: DNS (port 53) may only go through the TUN. Windows sends a name
// query to the DNS servers of all interfaces, not only to those of the TUN:
// to the first server of each interface, then to all of them when no answer
// arrives within a second or two. It sends the queries for the servers of
// an interface out through that interface, whatever the routes say, and
// other programs reach an on-link resolver, like 192.168.1.1 from DHCP,
// through its LAN route, which is more specific than the TUN's default
// route. Since Windows 11 and Server 2022, Windows may also send its
// queries over HTTPS or TLS, so there its DNS Client service may not
// - ports: connections to these TCP or UDP ports may only go through the
// TUN, those of DNS (53), DNS over TLS (TCP 853) and DNS over QUIC (UDP
// 853) with "dns". Windows sends a name query to the DNS servers of all
// interfaces, not only to those of the TUN: to the first server of each
// interface, then to all of them when no answer arrives within a second or
// two. It sends the queries for the servers of an interface out through
// that interface, whatever the routes say, and other programs reach an
// on-link resolver, like 192.168.1.1 from DHCP, through its LAN route,
// which is more specific than the TUN's default route.
// - dnsClient: since Windows 11 and Server 2022, Windows may also send its
// queries over HTTPS or TLS, to any port, so its DNS Client service may not
// connect outside the TUN at all, except for name resolution on the local
// link (mDNS, LLMNR).
// - ipv4, ipv6: no IPv4, or no IPv6, at all, in either direction, for a TUN
@@ -196,7 +257,7 @@ func condition(field *windows.GUID, typ uint32, value uintptr) fwpmFilterConditi
//
// The filters live in a dynamic WFP session: closing the returned engine handle
// with closeWFPEngine deletes them, and so does Windows when the process dies.
func blockLeaks(tun winipcfg.LUID, dns, ipv4, ipv6 bool) (windows.Handle, error) {
func blockLeaks(tun winipcfg.LUID, block leakFilters) (windows.Handle, error) {
engine, err := openWFPEngine()
if err != nil {
return 0, err
@@ -205,7 +266,7 @@ func blockLeaks(tun winipcfg.LUID, dns, ipv4, ipv6 bool) (windows.Handle, error)
closeWFPEngine(engine)
return 0, errors.New("FwpmTransactionBegin0 failed").Base(err)
}
err = addLeakFilters(engine, tun, dns, ipv4, ipv6)
err = addLeakFilters(engine, tun, block)
if err == nil {
if err = fwpmResult(procFwpmTransactionCommit0.Call(uintptr(engine))); err != nil {
err = errors.New("FwpmTransactionCommit0 failed").Base(err)
@@ -242,7 +303,7 @@ func closeWFPEngine(engine windows.Handle) {
// addLeakFilters adds the filters of blockLeaks in a sublayer of their own.
// blockLeaks runs it in a transaction, so that they take effect all at once.
func addLeakFilters(engine windows.Handle, tun winipcfg.LUID, dns, ipv4, ipv6 bool) error {
func addLeakFilters(engine windows.Handle, tun winipcfg.LUID, block leakFilters) error {
exe, err := os.Executable()
if err != nil {
return err
@@ -285,51 +346,55 @@ func addLeakFilters(engine windows.Handle, tun winipcfg.LUID, dns, ipv4, ipv6 bo
// hard permit. Connections from outside to Xray get an ordinary one, so
// that firewalls keep guarding its inbounds.
self := condition(&fwpmConditionALEAppID, fwpByteBlobType, uintptr(unsafe.Pointer(appID)))
dns53 := condition(&fwpmConditionIPRemotePort, fwpUint16, 53)
// DNS goes through the TUN when its local address is the TUN's, and it
// also leaves, or arrives, through the TUN. The local address alone
// A connection goes through the TUN when its local address is the TUN's,
// and it also leaves, or arrives, through the TUN. The local address alone
// decides by default, but with weak host sending or receiving enabled,
// packets of the TUN's address can use other interfaces. (The next hop,
// the interface replies would leave by, is not known for arriving ones.)
onTUN := func(field *windows.GUID) fwpmFilterCondition0 {
return condition(field, fwpUint64, uintptr(unsafe.Pointer(tunLUID)))
}
out := []fwpmFilterCondition0{dns53, onTUN(&fwpmConditionIPLocalInterface), onTUN(&fwpmConditionIPNexthopInterface)}
in := []fwpmFilterCondition0{dns53, onTUN(&fwpmConditionIPLocalInterface), onTUN(&fwpmConditionIPArrivalInterface)}
for _, layer := range []struct {
key *windows.GUID
selfFlags uint32
throughTUN []fwpmFilterCondition0
key *windows.GUID
selfFlags uint32
via *windows.GUID // the interface that has to be the TUN as well
}{
{&fwpmLayerALEAuthConnectV4, fwpmFilterFlagClearActionRight, out},
{&fwpmLayerALEAuthRecvAcceptV4, 0, in},
{&fwpmLayerALEAuthConnectV6, fwpmFilterFlagClearActionRight, out},
{&fwpmLayerALEAuthRecvAcceptV6, 0, in},
{&fwpmLayerALEAuthConnectV4, fwpmFilterFlagClearActionRight, &fwpmConditionIPNexthopInterface},
{&fwpmLayerALEAuthRecvAcceptV4, 0, &fwpmConditionIPArrivalInterface},
{&fwpmLayerALEAuthConnectV6, fwpmFilterFlagClearActionRight, &fwpmConditionIPNexthopInterface},
{&fwpmLayerALEAuthRecvAcceptV6, 0, &fwpmConditionIPArrivalInterface},
} {
if err := add(layer.key, "permit Xray", layer.selfFlags, fwpActionPermit, 4, self); err != nil {
return err
}
if dns {
if err := add(layer.key, "permit DNS through the TUN", 0, fwpActionPermit, 3, layer.throughTUN...); err != nil {
for _, port := range block.ports {
protocol := uintptr(windows.IPPROTO_UDP)
if port.tcp {
protocol = windows.IPPROTO_TCP
}
toPort := []fwpmFilterCondition0{
condition(&fwpmConditionIPProtocol, fwpUint8, protocol),
condition(&fwpmConditionIPRemotePort, fwpUint16, uintptr(port.port)),
}
throughTUN := slices.Concat(toPort, []fwpmFilterCondition0{onTUN(&fwpmConditionIPLocalInterface), onTUN(layer.via)})
if err := add(layer.key, "permit "+port.String()+" through the TUN", 0, fwpActionPermit, 3, throughTUN...); err != nil {
return err
}
if err := add(layer.key, "block DNS", 0, fwpActionBlock, 2, dns53); err != nil {
if err := add(layer.key, "block "+port.String(), 0, fwpActionBlock, 2, toPort...); err != nil {
return err
}
}
}
// Since Windows 11 and Server 2022 (build 20348), the DNS Client service
// may also send the queries for an interface's servers over HTTPS or TLS,
// out through that interface and to any port. So there it may only
// connect through the TUN, except for mDNS and LLMNR, which stay on the
// local link (over an IP version only while it is not blocked altogether).
// Earlier versions only query port 53, and may run the service in one
// process with others, which the filters would catch as well. Like
// Windows Firewall's rules for it, they recognize the service by its SID,
// Since Windows 11 and Server 2022, the DNS Client service may also send
// the queries for an interface's servers over HTTPS or TLS, out through
// that interface and to any port. So there it may only connect through
// the TUN, except for mDNS and LLMNR, which stay on the local link (over
// an IP version only while it is not blocked altogether). Like Windows
// Firewall's rules for it, the filters recognize the service by its SID,
// which Windows puts in the token of its process: the security descriptor
// grants that SID the right to match (FWP_ACTRL_MATCH_FILTER, CC in SDDL).
if _, _, build := windows.RtlGetNtVersionNumbers(); dns && build >= 20348 {
if block.dnsClient {
sd, err := windows.SecurityDescriptorFromString("O:SYG:SYD:(A;;CCRC;;;" + dnsClientSID + ")")
if err != nil {
return err
@@ -343,8 +408,8 @@ func addLeakFilters(engine windows.Handle, tun winipcfg.LUID, dns, ipv4, ipv6 bo
key *windows.GUID
localLink bool
}{
{&fwpmLayerALEAuthConnectV4, !ipv4},
{&fwpmLayerALEAuthConnectV6, !ipv6},
{&fwpmLayerALEAuthConnectV4, !block.ipv4},
{&fwpmLayerALEAuthConnectV6, !block.ipv6},
} {
if err := add(layer.key, "permit the DNS Client service through the TUN", 0, fwpActionPermit, 3, dnsClient, onTUN(&fwpmConditionIPLocalInterface), onTUN(&fwpmConditionIPNexthopInterface)); err != nil {
return err
@@ -367,7 +432,7 @@ func addLeakFilters(engine windows.Handle, tun winipcfg.LUID, dns, ipv4, ipv6 bo
matchType: fwpMatchFlagsAllSet,
conditionValue: fwpValue0{typ: fwpUint32, value: fwpConditionFlagIsLoopback},
}
if ipv4 {
if block.ipv4 {
// DHCP keeps the addresses of the other interfaces, which Xray's own
// connections use.
dhcp := []fwpmFilterCondition0{
@@ -387,7 +452,7 @@ func addLeakFilters(engine windows.Handle, tun winipcfg.LUID, dns, ipv4, ipv6 bo
}
}
}
if ipv6 {
if block.ipv6 {
// Neighbor and multicast listener discovery, ICMPv6 130-137 and 143,
// whose type and code sit where the local and remote port are.
discovery := []fwpmFilterCondition0{condition(&fwpmConditionIPProtocol, fwpUint8, windows.IPPROTO_ICMPV6)}
+28 -1
View File
@@ -114,11 +114,38 @@ func TestLeakFiltersAccepted(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if err := addLeakFilters(engine, loopback, true, true, true); err != nil {
leaks, err := ExpandWfpBlockLeak([]string{"dns", "misconfigtun", "tcp/443"})
if err != nil {
t.Fatal(err)
}
block := newLeakFilters(leaks, false, false)
block.dnsClient = true // whatever the Windows version
if err := addLeakFilters(engine, loopback, block); err != nil {
skipUnlessElevated(err)
}
}
func TestNewLeakFilters(t *testing.T) {
leaks, err := ExpandWfpBlockLeak([]string{"dns", "misconfigtun"})
if err != nil {
t.Fatal(err)
}
block := newLeakFilters(leaks, true, false)
want := "udp/53, tcp/53, tcp/853, udp/853, IPv6"
if dnsClientUsesAnyPort() {
want = "udp/53, tcp/53, tcp/853, udp/853, windnsclient, IPv6"
}
if got := block.String(); got != want {
t.Errorf("String() = %q, want %q", got, want)
}
if !block.systemDNS() {
t.Error("systemDNS() = false with dns")
}
if block := newLeakFilters([]string{"tcp/853", "udp/853", "misconfigtun"}, true, true); block.String() != "tcp/853, udp/853" || block.systemDNS() {
t.Errorf("tcp/853, udp/853: %q, systemDNS() = %v", block.String(), block.systemDNS())
}
}
func TestDNSClientSID(t *testing.T) {
sid, _, _, err := windows.LookupSID("", `NT SERVICE\Dnscache`)
if err != nil {
+79
View File
@@ -0,0 +1,79 @@
package tun
import (
"slices"
"strconv"
"strings"
"github.com/xtls/xray-core/common/errors"
)
// dnsLeaks is what "dns" in autoSystemWfpBlockLeak stands for: DNS over UDP and
// TCP, Windows' DNS Client service, which may also use other protocols and
// ports, DNS over TLS, and DNS over QUIC.
var dnsLeaks = []string{"udp/53", "tcp/53", "windnsclient", "tcp/853", "udp/853"}
// leakPort is a "tcp/<port>" or "udp/<port>" value of autoSystemWfpBlockLeak:
// connections to that port may only go through the TUN.
type leakPort struct {
tcp bool
port uint16
}
func (p leakPort) String() string {
if p.tcp {
return "tcp/" + strconv.Itoa(int(p.port))
}
return "udp/" + strconv.Itoa(int(p.port))
}
// parseLeakPort parses a lower case "tcp/<port>" or "udp/<port>" value.
func parseLeakPort(value string) (leakPort, bool) {
protocol, port, ok := strings.Cut(value, "/")
if !ok || protocol != "tcp" && protocol != "udp" {
return leakPort{}, false
}
n, err := strconv.ParseUint(port, 10, 16)
if err != nil || n == 0 {
return leakPort{}, false
}
return leakPort{tcp: protocol == "tcp", port: uint16(n)}, true
}
// ExpandWfpBlockLeak checks the values of autoSystemWfpBlockLeak and returns
// them in lower case, in order and without duplicates, with "dns" replaced by
// what it stands for.
func ExpandWfpBlockLeak(values []string) ([]string, error) {
var leaks []string
add := func(leak string) {
if !slices.Contains(leaks, leak) {
leaks = append(leaks, leak)
}
}
for _, value := range values {
switch value = strings.ToLower(value); value {
case "dns":
for _, leak := range dnsLeaks {
add(leak)
}
case "windnsclient", "misconfigtun":
add(value)
default:
port, ok := parseLeakPort(value)
if !ok {
return nil, errors.New("unknown autoSystemWfpBlockLeak value: ", value)
}
add(port.String())
}
}
return leaks, nil
}
// BlocksSystemDNS tells whether expanded autoSystemWfpBlockLeak values keep the
// system's DNS (port 53, or Windows' DNS Client service) inside the TUN, which
// then needs DNS servers of its own.
func BlocksSystemDNS(leaks []string) bool {
return slices.ContainsFunc(leaks, func(leak string) bool {
return leak == "udp/53" || leak == "tcp/53" || leak == "windnsclient"
})
}
+40
View File
@@ -0,0 +1,40 @@
package tun
import (
"slices"
"testing"
)
func TestExpandWfpBlockLeak(t *testing.T) {
for _, c := range []struct {
values []string
leaks []string
systemDNS bool
}{
{nil, nil, false},
{[]string{"dns"}, []string{"udp/53", "tcp/53", "windnsclient", "tcp/853", "udp/853"}, true},
{[]string{"DNS", "MisconfigTUN"}, []string{"udp/53", "tcp/53", "windnsclient", "tcp/853", "udp/853", "misconfigtun"}, true},
{[]string{"tcp/853", "dns", "TCP/0853"}, []string{"tcp/853", "udp/53", "tcp/53", "windnsclient", "udp/853"}, true},
{[]string{"tcp/853", "udp/853", "misconfigtun"}, []string{"tcp/853", "udp/853", "misconfigtun"}, false},
{[]string{"udp/65535", "tcp/1"}, []string{"udp/65535", "tcp/1"}, false},
{[]string{"tcp/53"}, []string{"tcp/53"}, true},
{[]string{"windnsclient"}, []string{"windnsclient"}, true},
} {
leaks, err := ExpandWfpBlockLeak(c.values)
if err != nil {
t.Errorf("%v: %v", c.values, err)
continue
}
if !slices.Equal(leaks, c.leaks) {
t.Errorf("%v: got %v, want %v", c.values, leaks, c.leaks)
}
if got := BlocksSystemDNS(leaks); got != c.systemDNS {
t.Errorf("%v: BlocksSystemDNS() = %v", c.values, got)
}
}
for _, value := range []string{"", "ip", "tcp", "tcp/", "tcp/0", "udp/65536", "tcp/-1", "tcp/+53", "tcp/53/1", "udp/abc", "icmp/1", "tcp:53", " dns"} {
if _, err := ExpandWfpBlockLeak([]string{value}); err == nil {
t.Errorf("%q was accepted", value)
}
}
}
+82
View File
@@ -3,6 +3,8 @@ package scenarios
import (
"encoding/base64"
"encoding/hex"
"fmt"
"io"
"sync"
"testing"
"time"
@@ -20,6 +22,7 @@ import (
"github.com/xtls/xray-core/proxy/dokodemo"
"github.com/xtls/xray-core/proxy/freedom"
"github.com/xtls/xray-core/proxy/vless"
"github.com/xtls/xray-core/proxy/vless/encoding"
"github.com/xtls/xray-core/proxy/vless/inbound"
"github.com/xtls/xray-core/proxy/vless/outbound"
"github.com/xtls/xray-core/testing/servers/tcp"
@@ -657,3 +660,82 @@ func TestVlessRealityFingerprints(t *testing.T) {
}
wg.Wait()
}
// A blocked connection must be closed after blockDelay even while the client keeps it open and sends nothing.
func TestVlessBlockDelay(t *testing.T) {
tcpServer := tcp.Server{
MsgProcessor: xor,
}
dest, err := tcpServer.Start()
common.Must(err)
defer tcpServer.Close()
for _, delay := range []uint64{0, 1} {
t.Run(fmt.Sprint("blockDelay=", delay), func(t *testing.T) {
userID := protocol.NewID(uuid.New())
serverPort := tcp.PickPort()
serverConfig := &core.Config{
Inbound: []*core.InboundHandlerConfig{
{
ReceiverSettings: serial.ToTypedMessage(&proxyman.ReceiverConfig{
PortList: &net.PortList{Range: []*net.PortRange{net.SinglePortRange(serverPort)}},
Listen: net.NewIPOrDomain(net.LocalHostIP),
}),
ProxySettings: serial.ToTypedMessage(&inbound.Config{
Users: []*protocol.User{
{
Account: serial.ToTypedMessage(&vless.Account{
Id: userID.String(),
}),
},
},
}),
},
},
Outbound: []*core.OutboundHandlerConfig{
{
ProxySettings: serial.ToTypedMessage(&freedom.Config{
FinalRules: []*freedom.FinalRuleConfig{
{
Action: freedom.RuleAction_Block,
BlockDelay: &freedom.Range{Min: delay, Max: delay},
},
},
}),
},
},
}
servers, err := InitializeServerConfigs(serverConfig)
common.Must(err)
defer CloseAllServers(servers)
conn, err := net.DialTCP("tcp", nil, &net.TCPAddr{
IP: []byte{127, 0, 0, 1},
Port: int(serverPort),
})
common.Must(err)
defer conn.Close()
start := time.Now()
common.Must(encoding.EncodeRequestHeader(conn, &protocol.RequestHeader{
Version: encoding.Version,
User: &protocol.MemoryUser{Account: &vless.MemoryAccount{ID: userID}},
Command: protocol.RequestCommandTCP,
Address: dest.Address,
Port: dest.Port,
}, &encoding.Addons{}))
blockDelay := time.Duration(delay) * time.Second
common.Must(conn.SetReadDeadline(time.Now().Add(blockDelay + 5*time.Second)))
_, err = io.Copy(io.Discard, conn)
elapsed := time.Since(start)
if err, ok := err.(net.Error); ok && err.Timeout() {
t.Fatalf("connection is still open after %v", elapsed)
}
if elapsed < blockDelay {
t.Fatalf("connection is closed after %v, before blockDelay", elapsed)
}
})
}
}
+8 -8
View File
@@ -114,13 +114,13 @@ func applyOutboundSocketOptions(network string, address string, fd uintptr, conf
if config.TcpKeepAliveIdle > 0 || config.TcpKeepAliveInterval > 0 {
if config.TcpKeepAliveIdle > 0 {
if err := unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, unix.TCP_KEEPALIVE, int(config.TcpKeepAliveInterval)); err != nil {
return errors.New("failed to set TCP_KEEPINTVL", err)
if err := unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, unix.TCP_KEEPALIVE, int(config.TcpKeepAliveIdle)); err != nil {
return errors.New("failed to set TCP_KEEPALIVE", err)
}
}
if config.TcpKeepAliveInterval > 0 {
if err := unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, sysTCP_KEEPINTVL, int(config.TcpKeepAliveIdle)); err != nil {
return errors.New("failed to set TCP_KEEPIDLE", err)
if err := unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, sysTCP_KEEPINTVL, int(config.TcpKeepAliveInterval)); err != nil {
return errors.New("failed to set TCP_KEEPINTVL", err)
}
}
if err := unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_KEEPALIVE, 1); err != nil {
@@ -204,13 +204,13 @@ func applyInboundSocketOptions(network string, fd uintptr, config *SocketConfig)
if config.TcpKeepAliveIdle > 0 || config.TcpKeepAliveInterval > 0 {
if config.TcpKeepAliveIdle > 0 {
if err := unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, unix.TCP_KEEPALIVE, int(config.TcpKeepAliveInterval)); err != nil {
return errors.New("failed to set TCP_KEEPINTVL", err)
if err := unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, unix.TCP_KEEPALIVE, int(config.TcpKeepAliveIdle)); err != nil {
return errors.New("failed to set TCP_KEEPALIVE", err)
}
}
if config.TcpKeepAliveInterval > 0 {
if err := unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, sysTCP_KEEPINTVL, int(config.TcpKeepAliveIdle)); err != nil {
return errors.New("failed to set TCP_KEEPIDLE", err)
if err := unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, sysTCP_KEEPINTVL, int(config.TcpKeepAliveInterval)); err != nil {
return errors.New("failed to set TCP_KEEPINTVL", err)
}
}
if err := unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_KEEPALIVE, 1); err != nil {
+12
View File
@@ -2,6 +2,8 @@ package internet
import (
"context"
go_errors "errors"
"runtime"
"sync"
"syscall"
"time"
@@ -92,6 +94,16 @@ func (d *DefaultSystemDialer) Dial(ctx context.Context, src net.Address, dest ne
}
}
packetConn, err := lc.ListenPacket(ctx, srcAddr.Network(), srcAddr.String())
var bindErr *BindError
if err != nil && runtime.GOOS == "windows" && go_errors.As(err, &bindErr) {
// Keep dual-stack UDP when possible. If the interface cannot bind both
// families, retry a socket that cannot send on the unbound family.
network := "udp6"
if destAddr.IP.To4() != nil {
network = "udp4"
}
packetConn, err = lc.ListenPacket(ctx, network, srcAddr.String())
}
if err != nil {
return nil, err
}
@@ -0,0 +1,63 @@
package internet_test
import (
"context"
"errors"
"slices"
"syscall"
"testing"
"github.com/xtls/xray-core/common/net"
"github.com/xtls/xray-core/transport/internet"
"golang.org/x/sys/windows"
)
// TestDialUDPBindFallback checks that a UDP dial whose dual-stack socket
// cannot be bound to both IP versions, as on an interface without one of
// them, moves to a socket of the destination's version only, and still
// fails when that one cannot be bound either.
func TestDialUDPBindFallback(t *testing.T) {
for _, c := range []struct {
dest string
reject []string // the kinds of socket that cannot be bound
tried []string // the kinds of socket tried, in order
ok bool
}{
{"127.0.0.1", nil, []string{"dual"}, true},
{"127.0.0.1", []string{"dual"}, []string{"dual", "udp4"}, true},
{"::1", []string{"dual"}, []string{"dual", "udp6"}, true},
{"127.0.0.1", []string{"dual", "udp4"}, []string{"dual", "udp4"}, false},
} {
var tried []string
internet.ControllersLock.Lock()
saved := internet.Controllers
internet.Controllers = []func(string, string, syscall.RawConn) error{func(network, _ string, rc syscall.RawConn) error {
kind := network
if network == "udp6" {
rc.Control(func(fd uintptr) {
if v6only, _ := windows.GetsockoptInt(windows.Handle(fd), windows.IPPROTO_IPV6, windows.IPV6_V6ONLY); v6only == 0 {
kind = "dual"
}
})
}
tried = append(tried, kind)
if slices.Contains(c.reject, kind) {
return internet.NewBindError(errors.New("cannot bind " + kind))
}
return nil
}}
internet.ControllersLock.Unlock()
conn, err := internet.DialSystem(context.Background(), net.UDPDestination(net.ParseAddress(c.dest), 53), nil)
if conn != nil {
conn.Close()
}
internet.ControllersLock.Lock()
internet.Controllers = saved
internet.ControllersLock.Unlock()
if (err == nil) != c.ok || !slices.Equal(tried, c.tried) {
t.Errorf("%s rejecting %v: tried %v, err %v", c.dest, c.reject, tried, err)
}
}
}
@@ -46,6 +46,12 @@ func (c *connection) Read(b []byte) (int, error) {
for {
reader, err := c.getReader()
if err != nil {
// A close frame with a normal status (1000) or no status (1005) ends
// the byte stream the same way a FIN does, so report io.EOF instead
// of an error. Other close codes keep their error semantics.
if websocket.IsCloseError(err, websocket.CloseNormalClosure, websocket.CloseNoStatusReceived) {
return 0, io.EOF
}
return 0, err
}
+96 -11
View File
@@ -37,6 +37,7 @@ type Config struct {
HoleTimeoutMs uint32 `protobuf:"varint,11,opt,name=hole_timeout_ms,json=holeTimeoutMs,proto3" json:"hole_timeout_ms,omitempty"`
Template string `protobuf:"bytes,12,opt,name=template,proto3" json:"template,omitempty"`
Xmux *splithttp.XmuxConfig `protobuf:"bytes,13,opt,name=xmux,proto3" json:"xmux,omitempty"`
S3 *S3Config `protobuf:"bytes,14,opt,name=s3,proto3" json:"s3,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -162,11 +163,86 @@ func (x *Config) GetXmux() *splithttp.XmuxConfig {
return nil
}
func (x *Config) GetS3() *S3Config {
if x != nil {
return x.S3
}
return nil
}
type S3Config struct {
state protoimpl.MessageState `protogen:"open.v1"`
Endpoint string `protobuf:"bytes,1,opt,name=endpoint,proto3" json:"endpoint,omitempty"`
Region string `protobuf:"bytes,2,opt,name=region,proto3" json:"region,omitempty"`
Bucket string `protobuf:"bytes,3,opt,name=bucket,proto3" json:"bucket,omitempty"`
PathStyle bool `protobuf:"varint,4,opt,name=path_style,json=pathStyle,proto3" json:"path_style,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *S3Config) Reset() {
*x = S3Config{}
mi := &file_transport_internet_xdrive_config_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *S3Config) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*S3Config) ProtoMessage() {}
func (x *S3Config) ProtoReflect() protoreflect.Message {
mi := &file_transport_internet_xdrive_config_proto_msgTypes[1]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use S3Config.ProtoReflect.Descriptor instead.
func (*S3Config) Descriptor() ([]byte, []int) {
return file_transport_internet_xdrive_config_proto_rawDescGZIP(), []int{1}
}
func (x *S3Config) GetEndpoint() string {
if x != nil {
return x.Endpoint
}
return ""
}
func (x *S3Config) GetRegion() string {
if x != nil {
return x.Region
}
return ""
}
func (x *S3Config) GetBucket() string {
if x != nil {
return x.Bucket
}
return ""
}
func (x *S3Config) GetPathStyle() bool {
if x != nil {
return x.PathStyle
}
return false
}
var File_transport_internet_xdrive_config_proto protoreflect.FileDescriptor
const file_transport_internet_xdrive_config_proto_rawDesc = "" +
"\n" +
"&transport/internet/xdrive/config.proto\x12\x1exray.transport.internet.xdrive\x1a)transport/internet/splithttp/config.proto\"\x8e\x04\n" +
"&transport/internet/xdrive/config.proto\x12\x1exray.transport.internet.xdrive\x1a)transport/internet/splithttp/config.proto\"\xc8\x04\n" +
"\x06Config\x12#\n" +
"\rremote_folder\x18\x01 \x01(\tR\fremoteFolder\x12\x18\n" +
"\aservice\x18\x02 \x01(\tR\aservice\x12\x18\n" +
@@ -181,7 +257,14 @@ const file_transport_internet_xdrive_config_proto_rawDesc = "" +
" \x01(\rR\reagerWindowMs\x12&\n" +
"\x0fhole_timeout_ms\x18\v \x01(\rR\rholeTimeoutMs\x12\x1a\n" +
"\btemplate\x18\f \x01(\tR\btemplate\x12A\n" +
"\x04xmux\x18\r \x01(\v2-.xray.transport.internet.splithttp.XmuxConfigR\x04xmuxB5Z3github.com/xtls/xray-core/transport/internet/xdriveb\x06proto3"
"\x04xmux\x18\r \x01(\v2-.xray.transport.internet.splithttp.XmuxConfigR\x04xmux\x128\n" +
"\x02s3\x18\x0e \x01(\v2(.xray.transport.internet.xdrive.S3ConfigR\x02s3\"u\n" +
"\bS3Config\x12\x1a\n" +
"\bendpoint\x18\x01 \x01(\tR\bendpoint\x12\x16\n" +
"\x06region\x18\x02 \x01(\tR\x06region\x12\x16\n" +
"\x06bucket\x18\x03 \x01(\tR\x06bucket\x12\x1d\n" +
"\n" +
"path_style\x18\x04 \x01(\bR\tpathStyleB5Z3github.com/xtls/xray-core/transport/internet/xdriveb\x06proto3"
var (
file_transport_internet_xdrive_config_proto_rawDescOnce sync.Once
@@ -195,18 +278,20 @@ func file_transport_internet_xdrive_config_proto_rawDescGZIP() []byte {
return file_transport_internet_xdrive_config_proto_rawDescData
}
var file_transport_internet_xdrive_config_proto_msgTypes = make([]protoimpl.MessageInfo, 1)
var file_transport_internet_xdrive_config_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
var file_transport_internet_xdrive_config_proto_goTypes = []any{
(*Config)(nil), // 0: xray.transport.internet.xdrive.Config
(*splithttp.XmuxConfig)(nil), // 1: xray.transport.internet.splithttp.XmuxConfig
(*S3Config)(nil), // 1: xray.transport.internet.xdrive.S3Config
(*splithttp.XmuxConfig)(nil), // 2: xray.transport.internet.splithttp.XmuxConfig
}
var file_transport_internet_xdrive_config_proto_depIdxs = []int32{
1, // 0: xray.transport.internet.xdrive.Config.xmux:type_name -> xray.transport.internet.splithttp.XmuxConfig
1, // [1:1] is the sub-list for method output_type
1, // [1:1] is the sub-list for method input_type
1, // [1:1] is the sub-list for extension type_name
1, // [1:1] is the sub-list for extension extendee
0, // [0:1] is the sub-list for field type_name
2, // 0: xray.transport.internet.xdrive.Config.xmux:type_name -> xray.transport.internet.splithttp.XmuxConfig
1, // 1: xray.transport.internet.xdrive.Config.s3:type_name -> xray.transport.internet.xdrive.S3Config
2, // [2:2] is the sub-list for method output_type
2, // [2:2] is the sub-list for method input_type
2, // [2:2] is the sub-list for extension type_name
2, // [2:2] is the sub-list for extension extendee
0, // [0:2] is the sub-list for field type_name
}
func init() { file_transport_internet_xdrive_config_proto_init() }
@@ -220,7 +305,7 @@ func file_transport_internet_xdrive_config_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_transport_internet_xdrive_config_proto_rawDesc), len(file_transport_internet_xdrive_config_proto_rawDesc)),
NumEnums: 0,
NumMessages: 1,
NumMessages: 2,
NumExtensions: 0,
NumServices: 0,
},
+8
View File
@@ -19,4 +19,12 @@ message Config {
uint32 hole_timeout_ms = 11;
string template = 12;
xray.transport.internet.splithttp.XmuxConfig xmux = 13;
S3Config s3 = 14;
}
message S3Config {
string endpoint = 1;
string region = 2;
string bucket = 3;
bool path_style = 4;
}
+24 -4
View File
@@ -2,6 +2,7 @@ package xdrive
import (
"context"
"io"
"os"
"sync"
"time"
@@ -11,10 +12,12 @@ import (
var placeholderAddr = &net.TCPAddr{IP: net.IP{127, 0, 0, 1}, Port: 0}
// Conn is a session in segments, walWriter and walReader, or in slots,
// slotWriter and slotReader
type Conn struct {
cancel context.CancelFunc
writer *walWriter
reader *walReader
writer io.WriteCloser
reader chunkReader
onClose func()
readBuf []byte
@@ -27,6 +30,13 @@ type Conn struct {
closeErr error
}
type chunkReader interface {
// chunks is closed when there are no more, then Err says why
chunks() <-chan []byte
Err() error
Wake()
}
func newConn(ctx context.Context, storage Storage, writePrefix, readPrefix string, p params, onClose func()) *Conn {
ctx, cancel := context.WithCancel(ctx)
return &Conn{
@@ -37,6 +47,16 @@ func newConn(ctx context.Context, storage Storage, writePrefix, readPrefix strin
}
}
func newSlotConn(ctx context.Context, store slotStore, write, read slotTable, p params, onClose func()) *Conn {
ctx, cancel := context.WithCancel(ctx)
return &Conn{
cancel: cancel,
writer: newSlotWriter(ctx, store, write, p),
reader: newSlotReader(ctx, store, read, p),
onClose: onClose,
}
}
func (c *Conn) Read(b []byte) (int, error) {
if len(c.readBuf) == 0 {
data, err := c.receive()
@@ -53,7 +73,7 @@ func (c *Conn) Read(b []byte) (int, error) {
func (c *Conn) receive() ([]byte, error) {
deadline := c.getDeadline(true)
if deadline.IsZero() {
data, ok := <-c.reader.ch
data, ok := <-c.reader.chunks()
if !ok {
return nil, c.reader.Err()
}
@@ -67,7 +87,7 @@ func (c *Conn) receive() ([]byte, error) {
defer timer.Stop()
select {
case data, ok := <-c.reader.ch:
case data, ok := <-c.reader.chunks():
if !ok {
return nil, c.reader.Err()
}
+106 -46
View File
@@ -401,24 +401,30 @@ func (s *driveStorage) Put(ctx context.Context, name string, data []byte) error
}
func (s *driveStorage) putInline(ctx context.Context, name string, data []byte) error {
_, err := s.create(ctx, name, base64.StdEncoding.EncodeToString(data))
return err
}
// create makes an object with description and no body and returns its id
func (s *driveStorage) create(ctx context.Context, name, description string) (string, error) {
flat := flatten(name)
body, err := json.Marshal(map[string]interface{}{
"name": flat,
"parents": []string{s.folder},
"description": base64.StdEncoding.EncodeToString(data),
"description": description,
})
if err != nil {
return errors.New("failed to build the inline metadata").Base(err)
return "", errors.New("failed to build the inline metadata").Base(err)
}
status, payload, err := s.do(ctx, http.MethodPost, s.filesURL+"?fields=id",
"application/json; charset=UTF-8", body)
if err != nil {
return err
return "", err
}
if status != http.StatusOK {
return errors.New("Drive rejected the inline upload of ", name,
return "", errors.New("Drive rejected the inline upload of ", name,
" with ", status, ": ", string(payload))
}
@@ -426,12 +432,23 @@ func (s *driveStorage) putInline(ctx context.Context, name string, data []byte)
ID string `json:"id"`
}
if err := json.Unmarshal(payload, &parsed); err != nil {
return errors.New("failed to parse the Drive upload response").Base(err)
return "", errors.New("failed to parse the Drive upload response").Base(err)
}
if parsed.ID != "" {
s.rememberID(flat, parsed.ID)
}
return nil
return parsed.ID, nil
}
// multipartBody is a body of an upload with metadata and data
func multipartBody(metadata, data []byte) []byte {
var body bytes.Buffer
fmt.Fprintf(&body, "--%s\r\nContent-Type: application/json; charset=UTF-8\r\n\r\n", driveBoundary)
body.Write(metadata)
fmt.Fprintf(&body, "\r\n--%s\r\nContent-Type: application/octet-stream\r\n\r\n", driveBoundary)
body.Write(data)
fmt.Fprintf(&body, "\r\n--%s--\r\n", driveBoundary)
return body.Bytes()
}
func (s *driveStorage) putMedia(ctx context.Context, name string, data []byte) error {
@@ -445,15 +462,8 @@ func (s *driveStorage) putMedia(ctx context.Context, name string, data []byte) e
return errors.New("failed to build the upload metadata").Base(err)
}
var body bytes.Buffer
fmt.Fprintf(&body, "--%s\r\nContent-Type: application/json; charset=UTF-8\r\n\r\n", driveBoundary)
body.Write(metadata)
fmt.Fprintf(&body, "\r\n--%s\r\nContent-Type: application/octet-stream\r\n\r\n", driveBoundary)
body.Write(data)
fmt.Fprintf(&body, "\r\n--%s--\r\n", driveBoundary)
status, payload, err := s.do(ctx, http.MethodPost, s.uploadURL,
"multipart/related; boundary="+driveBoundary, body.Bytes())
"multipart/related; boundary="+driveBoundary, multipartBody(metadata, data))
if err != nil {
return err
}
@@ -480,45 +490,22 @@ func (s *driveStorage) Get(ctx context.Context, name string) ([]byte, error) {
return nil, err
}
status, payload, err := s.do(ctx, http.MethodGet,
s.filesURL+"/"+url.PathEscape(id)+"?alt=media&supportsAllDrives=true", "", nil)
if err != nil {
return nil, err
}
switch status {
case http.StatusOK:
if len(payload) > 0 {
return payload, nil
}
return s.getInline(ctx, id)
case http.StatusNotFound:
payload, err := s.readBody(ctx, id)
if err == errNotFound {
s.forgetID(flat)
return nil, errNotFound
default:
return nil, errors.New("Drive download of ", name, " answered ", status, ": ", string(payload))
}
if err != nil || len(payload) > 0 {
return payload, err
}
return s.getInline(ctx, id)
}
func (s *driveStorage) getInline(ctx context.Context, id string) ([]byte, error) {
status, payload, err := s.do(ctx, http.MethodGet,
s.filesURL+"/"+url.PathEscape(id)+"?fields=description&supportsAllDrives=true", "", nil)
if err != nil {
description, err := s.readDescription(ctx, id)
if err != nil || description == "" {
return nil, err
}
if status != http.StatusOK {
return nil, errors.New("Drive answered ", status, " for inline data: ", string(payload))
}
var parsed struct {
Description string `json:"description"`
}
if err := json.Unmarshal(payload, &parsed); err != nil {
return nil, errors.New("failed to parse the inline data").Base(err)
}
if parsed.Description == "" {
return nil, nil
}
data, err := base64.StdEncoding.DecodeString(parsed.Description)
data, err := base64.StdEncoding.DecodeString(description)
if err != nil {
return nil, errors.New("the inline data is not valid base64").Base(err)
}
@@ -605,3 +592,76 @@ func (s *driveStorage) Close() error {
}
return nil
}
// makeSlot implements slotStore
func (s *driveStorage) makeSlot(ctx context.Context, name string) (string, error) {
id, err := s.create(ctx, name, "")
if err == nil && id == "" {
err = errors.New("Drive returned no id for ", name)
}
return id, err
}
// setSlot implements slotStore, a description and a body go in one request
func (s *driveStorage) setSlot(ctx context.Context, id, description string, body []byte) error {
metadata, err := json.Marshal(map[string]string{"description": description})
if err != nil {
return errors.New("failed to build the slot metadata").Base(err)
}
target := s.filesURL + "/" + url.PathEscape(id) + "?fields=id&supportsAllDrives=true"
contentType := "application/json; charset=UTF-8"
if body != nil {
upload, _, _ := strings.Cut(s.uploadURL, "?")
target = upload + "/" + url.PathEscape(id) + "?uploadType=multipart&fields=id&supportsAllDrives=true"
contentType = "multipart/related; boundary=" + driveBoundary
metadata = multipartBody(metadata, body)
}
status, payload, err := s.do(ctx, http.MethodPatch, target, contentType, metadata)
switch {
case err != nil:
return err
case status == http.StatusNotFound:
return errNotFound
case status != http.StatusOK:
return errors.New("Drive answered ", status, " for the update of ", id, ": ", string(payload))
}
return nil
}
// readDescription implements slotStore
func (s *driveStorage) readDescription(ctx context.Context, id string) (string, error) {
status, payload, err := s.do(ctx, http.MethodGet,
s.filesURL+"/"+url.PathEscape(id)+"?fields=description&supportsAllDrives=true", "", nil)
switch {
case err != nil:
return "", err
case status == http.StatusNotFound:
return "", errNotFound
case status != http.StatusOK:
return "", errors.New("Drive answered ", status, " for the description of ", id, ": ", string(payload))
}
var parsed struct {
Description string `json:"description"`
}
if err := json.Unmarshal(payload, &parsed); err != nil {
return "", errors.New("failed to parse the description of ", id).Base(err)
}
return parsed.Description, nil
}
// readBody implements slotStore
func (s *driveStorage) readBody(ctx context.Context, id string) ([]byte, error) {
status, payload, err := s.do(ctx, http.MethodGet,
s.filesURL+"/"+url.PathEscape(id)+"?alt=media&supportsAllDrives=true", "", nil)
switch {
case err != nil:
return nil, err
case status == http.StatusNotFound:
return nil, errNotFound
case status != http.StatusOK:
return nil, errors.New("Drive answered ", status, " for the body of ", id, ": ", string(payload))
}
return payload, nil
}
+5 -5
View File
@@ -57,7 +57,7 @@ func liveDriveConfig(t *testing.T) *Config {
config.Concurrency = uint32(envInt(t, "XRAY_XDRIVE_LIVE_CONCURRENCY"))
}
storage, err := newDriveStorage(nil, config)
storage, err := newDriveStorage(&internet.MemoryStreamConfig{}, config)
if err != nil {
t.Fatalf("newDriveStorage: %v", err)
}
@@ -83,7 +83,7 @@ func envInt(t *testing.T, name string) int {
func TestLiveDriveStorage(t *testing.T) {
config := liveDriveConfig(t)
storage, err := newDriveStorage(nil, config)
storage, err := newDriveStorage(&internet.MemoryStreamConfig{}, config)
if err != nil {
t.Fatalf("newDriveStorage: %v", err)
}
@@ -194,7 +194,7 @@ func TestLiveDriveTransport(t *testing.T) {
func TestLiveDriveParallelPut(t *testing.T) {
config := liveDriveConfig(t)
storage, err := newDriveStorage(nil, config)
storage, err := newDriveStorage(&internet.MemoryStreamConfig{}, config)
if err != nil {
t.Fatalf("newDriveStorage: %v", err)
}
@@ -266,7 +266,7 @@ func TestLiveDriveParallelPut(t *testing.T) {
func TestLiveDriveSegmentSweep(t *testing.T) {
config := liveDriveConfig(t)
storage, err := newDriveStorage(nil, config)
storage, err := newDriveStorage(&internet.MemoryStreamConfig{}, config)
if err != nil {
t.Fatalf("newDriveStorage: %v", err)
}
@@ -303,7 +303,7 @@ func TestLiveDriveSegmentSweep(t *testing.T) {
func TestLiveDriveListLag(t *testing.T) {
config := liveDriveConfig(t)
storage, err := newDriveStorage(nil, config)
storage, err := newDriveStorage(&internet.MemoryStreamConfig{}, config)
if err != nil {
t.Fatalf("newDriveStorage: %v", err)
}
+41 -4
View File
@@ -63,6 +63,7 @@ func newFakeDrive(t *testing.T) *fakeDrive {
}
mux.HandleFunc("/token", record(drive.handleToken))
mux.HandleFunc("/upload", record(drive.handleUpload))
mux.HandleFunc("/upload/", record(drive.handleUpload))
mux.HandleFunc("/files", record(drive.handleFiles))
mux.HandleFunc("/files/", record(drive.handleFile))
drive.server = httptest.NewServer(mux)
@@ -119,7 +120,8 @@ func (d *fakeDrive) handleUpload(w http.ResponseWriter, r *http.Request) {
return
}
var metadata struct {
Name string `json:"name"`
Name string `json:"name"`
Description string `json:"description"`
}
if err := json.NewDecoder(metaPart).Decode(&metadata); err != nil {
w.WriteHeader(http.StatusBadRequest)
@@ -131,6 +133,22 @@ func (d *fakeDrive) handleUpload(w http.ResponseWriter, r *http.Request) {
data, _ = io.ReadAll(dataPart)
}
if id, ok := strings.CutPrefix(r.URL.Path, "/upload/"); ok {
// an update of the body and description of a file
d.mu.Lock()
file := d.files[id]
if file != nil {
file.data, file.description = data, metadata.Description
}
d.mu.Unlock()
if file == nil || r.Method != http.MethodPatch {
w.WriteHeader(http.StatusNotFound)
return
}
json.NewEncoder(w).Encode(map[string]string{"id": id})
return
}
d.mu.Lock()
d.nextID++
id := fmt.Sprintf("id-%d", d.nextID)
@@ -223,6 +241,11 @@ func (d *fakeDrive) handleFile(w http.ResponseWriter, r *http.Request) {
if ok && r.Method == http.MethodDelete {
delete(d.files, id)
}
var snapshot fakeFile
if ok {
// a file can be updated, its fields are read here
snapshot = *file
}
d.mu.Unlock()
if !ok {
@@ -233,11 +256,25 @@ func (d *fakeDrive) handleFile(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
return
}
if strings.Contains(r.URL.RawQuery, "fields=description") {
json.NewEncoder(w).Encode(map[string]string{"description": file.description})
if r.Method == http.MethodPatch {
var meta struct {
Description string `json:"description"`
}
if err := json.NewDecoder(r.Body).Decode(&meta); err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
d.mu.Lock()
file.description = meta.Description
d.mu.Unlock()
json.NewEncoder(w).Encode(map[string]string{"id": id})
return
}
w.Write(file.data)
if strings.Contains(r.URL.RawQuery, "fields=description") {
json.NewEncoder(w).Encode(map[string]string{"description": snapshot.description})
return
}
w.Write(snapshot.data)
}
func (d *fakeDrive) shouldFail(kind string) bool {
+518
View File
@@ -0,0 +1,518 @@
package xdrive
import (
"bytes"
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/xml"
stderrors "errors"
"io"
"net"
"net/http"
"net/url"
"strconv"
"strings"
"sync"
"time"
"github.com/xtls/xray-core/common/errors"
"github.com/xtls/xray-core/transport/internet"
"golang.org/x/net/http/httpguts"
)
type s3Storage struct {
*s3Shared
client *http.Client
inflight chan struct{}
release func()
}
type s3Shared struct {
conns *connPool
base url.URL
prefix string
region string
accessKey, secretKey string
token string
backoff time.Duration
}
func s3Region(region string) string {
if region == "" {
return "us-east-1"
}
return region
}
func ValidateS3Config(c *Config) error {
if c == nil || c.S3 == nil {
return errors.New(`service "s3" needs an "s3" object`)
}
u, err := url.Parse(c.S3.Endpoint)
if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Hostname() == "" ||
u.User != nil || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || u.Opaque != "" {
return errors.New("S3 endpoint must be an http(s) URL without credentials, query or fragment")
}
if !httpguts.ValidHeaderFieldValue(c.S3.Region) || strings.ContainsAny(c.S3.Region, "/, \t") {
return errors.New("invalid S3 signing region")
}
bucket := c.S3.Bucket
if !validS3Name(bucket) || strings.Contains(bucket, "/") {
return errors.New("S3 needs a bucket name without path separators")
}
if !c.S3.PathStyle {
if net.ParseIP(u.Hostname()) != nil {
return errors.New("S3 virtual-hosted addressing requires a DNS endpoint")
}
// In this mode the bucket is part of the HTTP authority, not its path.
if strings.Trim(bucket, "abcdefghijklmnopqrstuvwxyz0123456789-.") != "" || strings.Contains(bucket, "..") ||
strings.HasPrefix(bucket, ".") || strings.HasSuffix(bucket, ".") {
return errors.New("S3 virtual-hosted addressing requires a DNS-compatible bucket name")
}
}
if !validS3Name(strings.TrimSuffix(c.RemoteFolder, "/")) {
return errors.New("S3 remoteFolder must be a nonempty relative prefix without empty, '.' or '..' components")
}
if len(c.Secrets) < 2 || len(c.Secrets) > 3 || c.Secrets[0] == "" || c.Secrets[1] == "" {
return errors.New("S3 needs AccessKeyID, SecretAccessKey and optionally SessionToken in secrets")
}
for _, secret := range c.Secrets {
if !httpguts.ValidHeaderFieldValue(secret) {
return errors.New("invalid S3 secret")
}
}
if strings.ContainsAny(c.Secrets[0], "/ ,\t") {
return errors.New("invalid S3 AccessKeyID")
}
return nil
}
func validS3Name(name string) bool {
for _, ch := range name {
if ch < 0x20 || ch == 0x7f || ch == '\\' {
return false
}
}
for _, part := range strings.Split(name, "/") {
if part == "" || part == "." || part == ".." {
return false
}
}
return true
}
func newS3Storage(streamSettings *internet.MemoryStreamConfig, c *Config) (*s3Storage, error) {
if err := ValidateS3Config(c); err != nil {
return nil, err
}
base, _ := url.Parse(c.S3.Endpoint)
// Encode the endpoint prefix for SigV4 without turning escaped slashes into separators.
segments := strings.Split(base.EscapedPath(), "/")
for i, segment := range segments {
decoded, _ := url.PathUnescape(segment)
segments[i] = strings.ReplaceAll(s3EscapePath(decoded), "/", "%2F")
}
base.RawPath = strings.Join(segments, "/")
if !strings.HasSuffix(base.RawPath, "/") {
base.Path += "/"
base.RawPath += "/"
}
if c.S3.PathStyle {
base.Path += c.S3.Bucket + "/"
base.RawPath += s3EscapePath(c.S3.Bucket) + "/"
} else {
host := c.S3.Bucket + "." + base.Hostname()
if base.Port() != "" {
host = net.JoinHostPort(host, base.Port())
}
base.Host = host
}
if streamSettings == nil {
streamSettings = &internet.MemoryStreamConfig{}
}
h1Hosts := new(sync.Map)
newClient := func(ctx context.Context) *http.Client {
client := newServiceClient(ctx, h1Hosts, streamSettings, driveTimeout, driveMaxInflight)
// A redirect changes the signed host/path. Do not follow it with the old
// signature or send the session token to a different endpoint.
client.CheckRedirect = s3NoRedirect
return client
}
s := &s3Storage{
s3Shared: &s3Shared{
conns: newConnPool(c.Xmux, newClient),
base: *base,
prefix: strings.TrimSuffix(c.RemoteFolder, "/") + "/",
region: s3Region(c.S3.Region),
accessKey: c.Secrets[0],
secretKey: c.Secrets[1],
backoff: 200 * time.Millisecond,
},
client: newClient(context.Background()),
inflight: make(chan struct{}, driveMaxInflight),
}
if len(c.Secrets) == 3 {
s.token = c.Secrets[2]
}
return s, nil
}
func s3NoRedirect(_ *http.Request, _ []*http.Request) error { return http.ErrUseLastResponse }
func (s *s3Storage) session() Storage {
transport := &sessionTransport{pool: s.conns}
return &s3Storage{
s3Shared: s.s3Shared,
client: &http.Client{Transport: transport, CheckRedirect: s3NoRedirect},
inflight: make(chan struct{}, cap(s.inflight)),
release: transport.Close,
}
}
func (s *s3Storage) Close() error {
if s.release != nil {
s.release()
}
return nil
}
// SigV4 requires %20 for spaces, uppercase escapes, and unescaped key separators.
func s3EscapePath(value string) string {
const digits = "0123456789ABCDEF"
var out strings.Builder
for i := 0; i < len(value); i++ {
ch := value[i]
if ch >= 'a' && ch <= 'z' || ch >= 'A' && ch <= 'Z' || ch >= '0' && ch <= '9' ||
ch == '-' || ch == '_' || ch == '.' || ch == '~' || ch == '/' {
out.WriteByte(ch)
} else {
out.WriteByte('%')
out.WriteByte(digits[ch>>4])
out.WriteByte(digits[ch&15])
}
}
return out.String()
}
func s3Hash(body []byte) string {
sum := sha256.Sum256(body)
return hex.EncodeToString(sum[:])
}
func s3HMAC(key []byte, value string) []byte {
h := hmac.New(sha256.New, key)
h.Write([]byte(value))
return h.Sum(nil)
}
func (s *s3Storage) sign(req *http.Request, payloadHash string, now time.Time) {
stamp, date := now.UTC().Format("20060102T150405Z"), now.UTC().Format("20060102")
req.Header.Set("X-Amz-Date", stamp)
req.Header.Set("X-Amz-Content-Sha256", payloadHash)
headers := "host:" + req.URL.Host + "\nx-amz-content-sha256:" + payloadHash + "\nx-amz-date:" + stamp + "\n"
signed := "host;x-amz-content-sha256;x-amz-date"
if contentType := req.Header.Get("Content-Type"); contentType != "" {
headers = "content-type:" + contentType + "\n" + headers
signed = "content-type;" + signed
}
if s.token != "" {
req.Header.Set("X-Amz-Security-Token", s.token)
headers += "x-amz-security-token:" + strings.Join(strings.Fields(s.token), " ") + "\n"
signed += ";x-amz-security-token"
}
// Sign the escaped path exactly as sent.
canonical := req.Method + "\n" + req.URL.EscapedPath() + "\n" + req.URL.RawQuery + "\n" +
headers + "\n" + signed + "\n" + payloadHash
scope := date + "/" + s.region + "/s3/aws4_request"
toSign := "AWS4-HMAC-SHA256\n" + stamp + "\n" + scope + "\n" + s3Hash([]byte(canonical))
key := s3HMAC([]byte("AWS4"+s.secretKey), date)
key = s3HMAC(key, s.region)
key = s3HMAC(key, "s3")
key = s3HMAC(key, "aws4_request")
req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential="+s.accessKey+"/"+scope+
", SignedHeaders="+signed+", Signature="+hex.EncodeToString(s3HMAC(key, toSign)))
}
func (s *s3Storage) target(key string, query url.Values) string {
u := s.base
u.Path += key
u.RawPath += s3EscapePath(key)
// Parameters have fixed ASCII names and one value each; SigV4 uses %20 for spaces.
u.RawQuery = strings.ReplaceAll(query.Encode(), "+", "%20")
return u.String()
}
func (s *s3Storage) objectKey(name string) (string, error) {
if !validS3Name(name) {
return "", errors.New("invalid relative S3 object name")
}
key := s.prefix + name
if len(key) > 1024 {
return "", errors.New("S3 object key exceeds 1024 bytes")
}
return key, nil
}
// Retries use fresh signatures and hold a concurrency slot until the response body is read.
func (s *s3Storage) do(ctx context.Context, method, key string, query url.Values, body []byte) (int, []byte, error) {
payloadHash := s3Hash(body)
backoff := s.backoff
var lastErr error
var retryAfter time.Duration
for attempt := 0; attempt < driveMaxAttempts; attempt++ {
if attempt > 0 {
delay := max(jitter(backoff), retryAfter)
timer := time.NewTimer(min(delay, driveMaxBackoff))
select {
case <-ctx.Done():
timer.Stop()
return 0, nil, ctx.Err()
case <-timer.C:
}
backoff = min(backoff*2, driveMaxBackoff)
}
select {
case s.inflight <- struct{}{}:
case <-ctx.Done():
return 0, nil, ctx.Err()
}
status, payload, after, err := s.attempt(ctx, method, key, query, body, payloadHash)
<-s.inflight
if ctx.Err() != nil {
return 0, nil, ctx.Err()
}
if err != nil && (!retryS3Error(err) || status >= 300 && !retryableStatus(status)) {
return 0, nil, err
}
if err == nil && !retryableStatus(status) {
return status, payload, nil
}
lastErr, retryAfter = err, after
if lastErr == nil {
lastErr = s3ResponseError(method, status, payload)
}
}
return 0, nil, lastErr
}
func retryS3Error(err error) bool {
var dns *net.DNSError
if stderrors.As(err, &dns) {
return dns.IsTimeout || dns.IsTemporary
}
var op *net.OpError
var network net.Error
return stderrors.Is(err, io.EOF) || stderrors.Is(err, io.ErrUnexpectedEOF) || stderrors.Is(err, errStalled) ||
stderrors.As(err, &op) || stderrors.As(err, &network) && network.Timeout()
}
func (s *s3Storage) attempt(ctx context.Context, method, key string, query url.Values, body []byte, hash string) (int, []byte, time.Duration, error) {
req, err := http.NewRequestWithContext(ctx, method, s.target(key, query), bytes.NewReader(body))
if err != nil {
return 0, nil, 0, errors.New("failed to build S3 request")
}
if method == http.MethodPut {
req.Header.Set("Content-Type", "application/octet-stream")
}
s.sign(req, hash, time.Now())
resp, err := s.client.Do(req)
if err != nil {
if urlErr, ok := err.(*url.Error); ok {
err = urlErr.Err
}
return 0, nil, 0, errors.New("S3 request failed").Base(err)
}
defer resp.Body.Close()
limit := int64(maxSegmentBytes)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
limit = 64 << 10
}
payload, err := io.ReadAll(io.LimitReader(resp.Body, limit+1))
if err != nil {
return resp.StatusCode, nil, 0, errors.New("failed to read S3 response (HTTP ", resp.StatusCode, ")").Base(err)
}
if int64(len(payload)) > limit {
return 0, nil, 0, errors.New("S3 response exceeds size limit")
}
var after time.Duration
if raw := resp.Header.Get("Retry-After"); raw != "" {
if seconds, err := strconv.ParseInt(raw, 10, 32); err == nil && seconds > 0 {
after = min(time.Duration(seconds)*time.Second, driveMaxBackoff)
} else if at, err := http.ParseTime(raw); err == nil {
after = min(max(time.Until(at), 0), driveMaxBackoff)
}
}
return resp.StatusCode, payload, after, nil
}
func s3Missing(status int, payload []byte) bool {
if status != http.StatusNotFound {
return false
}
code := s3ErrorCode(payload)
return code == "NoSuchKey" || code == "NotFound"
}
func s3ErrorCode(payload []byte) string {
var response struct {
XMLName xml.Name `xml:"Error"`
Code string
}
if xml.Unmarshal(payload, &response) != nil {
return ""
}
return response.Code
}
func s3ResponseError(method string, status int, payload []byte) error {
if code := s3ErrorCode(payload); code != "" {
return errors.New("S3 ", method, " answered ", status, " (", strconv.Quote(code), ")")
}
return errors.New("S3 ", method, " answered ", status)
}
func (s *s3Storage) Put(ctx context.Context, name string, data []byte) error {
key, err := s.objectKey(name)
if err != nil {
return err
}
status, payload, err := s.do(ctx, http.MethodPut, key, nil, data)
if err != nil {
return err
}
if status != http.StatusOK {
return s3ResponseError("PUT", status, payload)
}
return nil
}
func (s *s3Storage) Get(ctx context.Context, name string) ([]byte, error) {
key, err := s.objectKey(name)
if err != nil {
return nil, err
}
status, payload, err := s.do(ctx, http.MethodGet, key, nil, nil)
if err != nil {
return nil, err
}
if s3Missing(status, payload) {
return nil, errNotFound
}
if status != http.StatusOK {
return nil, s3ResponseError("GET", status, payload)
}
return payload, nil
}
type s3ListPage struct {
XMLName xml.Name `xml:"ListBucketResult"`
IsTruncated bool
NextContinuationToken string
Contents []struct{ Key string }
CommonPrefixes []struct{ Prefix string }
}
func (s *s3Storage) list(ctx context.Context, prefix, delimiter string, visit func(*s3ListPage) error) error {
query := url.Values{"list-type": {"2"}, "prefix": {prefix}, "max-keys": {"1000"}}
if delimiter != "" {
query.Set("delimiter", delimiter)
}
seen := make(map[string]bool)
for {
status, payload, err := s.do(ctx, http.MethodGet, "", query, nil)
if err != nil {
return err
}
if status != http.StatusOK {
return s3ResponseError("LIST", status, payload)
}
var page s3ListPage
if err := xml.Unmarshal(payload, &page); err != nil {
return errors.New("invalid S3 list response")
}
// Validate all keys before deleting any objects.
for _, item := range page.Contents {
if !strings.HasPrefix(item.Key, prefix) {
return errors.New("S3 returned an object outside the requested prefix")
}
}
for _, item := range page.CommonPrefixes {
if !strings.HasPrefix(item.Prefix, prefix) {
return errors.New("S3 returned a directory outside the requested prefix")
}
}
if page.IsTruncated {
if page.NextContinuationToken == "" || seen[page.NextContinuationToken] {
return errors.New("S3 list pagination did not advance")
}
seen[page.NextContinuationToken] = true
}
if err := visit(&page); err != nil {
return err
}
if !page.IsTruncated {
return nil
}
query.Set("continuation-token", page.NextContinuationToken)
}
}
func (s *s3Storage) List(ctx context.Context, prefix string) ([]Entry, error) {
key, err := s.objectKey(prefix)
if err != nil {
return nil, err
}
key += "/"
seen := make(map[string]bool)
var entries []Entry
add := func(full string) {
rest := strings.TrimPrefix(full, key)
name, _, _ := strings.Cut(rest, "/")
if name != "" && validS3Name(name) && !seen[name] {
seen[name] = true
entries = append(entries, Entry{Name: name})
}
}
err = s.list(ctx, key, "/", func(page *s3ListPage) error {
for _, item := range page.Contents {
add(item.Key)
}
for _, item := range page.CommonPrefixes {
add(item.Prefix)
}
return nil
})
return entries, err
}
func (s *s3Storage) deleteObject(ctx context.Context, key string) error {
status, payload, err := s.do(ctx, http.MethodDelete, key, nil, nil)
if err != nil {
return err
}
if status != http.StatusNoContent && status != http.StatusOK && !s3Missing(status, payload) {
return s3ResponseError("DELETE", status, payload)
}
return nil
}
// The trailing "/" prevents deleting streams/abcd when removing streams/abc.
func (s *s3Storage) Delete(ctx context.Context, name string) error {
key, err := s.objectKey(name)
if err != nil {
return err
}
if err := s.deleteObject(ctx, key); err != nil {
return err
}
return s.list(ctx, key+"/", "", func(page *s3ListPage) error {
for _, item := range page.Contents {
if err := s.deleteObject(ctx, item.Key); err != nil {
return err
}
}
return nil
})
}
+758
View File
@@ -0,0 +1,758 @@
package xdrive
import (
"bytes"
"context"
"encoding/base64"
"encoding/binary"
"encoding/json"
"hash/crc32"
"io"
"strconv"
"sync"
"time"
"github.com/xtls/xray-core/common/errors"
)
// A session on a slotStore is carried in a ring of slots for each direction,
// objects that both peers know by id: segment n is written to slot n % depth
// and read from it, with nothing to list and no index to wait for. The reader
// of a direction keeps how far it has read in a watermark, and the writer
// reuses a slot only once that says what was in it before was taken
const (
slotMagic = "XDRVSL01"
slotHeaderLen = len(slotMagic) + 8 + 1 + 4 + 4
// slotInlineLimit is how large a segment is at most to go in the description
// of its slot, which takes one request less than a body
slotInlineLimit = 14 << 10
slotDepthLimit = 8
// inlineRampSegments segments of one size in a row double it, from
// slotInlineLimit up to segmentBytes
inlineRampSegments = 8
slotInline = 1
slotMedia = 2
slotEnd = 3
slotErr = 4
)
var (
// the head is asked for between slotHuntFloor and slotHuntCeiling, a segment
// behind it at about the pace that segments come
slotHuntFloor = 50 * time.Millisecond
slotHuntCeiling = 200 * time.Millisecond
slotTailMin = 150 * time.Millisecond
slotTailMax = 800 * time.Millisecond
slotFetchAhead = 6
slotAckRetry = 100 * time.Millisecond
slotWatermarkTTL = 250 * time.Millisecond
slotStallTimeout = 60 * time.Second
slotCloseGrace = 10 * time.Second
slotGoneTimeout = 60 * time.Second
)
// slotStore is a storage of objects that are written and read again by id,
// each with a description and a body
type slotStore interface {
makeSlot(ctx context.Context, name string) (string, error)
// setSlot sets the description of a slot and, unless body is nil, its body
setSlot(ctx context.Context, id, description string, body []byte) error
readDescription(ctx context.Context, id string) (string, error)
readBody(ctx context.Context, id string) ([]byte, error)
}
// slotFrame is what the description of a slot says: which segment it holds,
// and either the segment itself or the size and checksum of the body that does
type slotFrame struct {
seq int64
mode byte
size int
crc uint32
data []byte
}
func newSlotFrame(seq int64, mode byte, data []byte) slotFrame {
f := slotFrame{seq: seq, mode: mode, size: len(data), crc: crc32.ChecksumIEEE(data)}
if mode == slotInline {
f.data = data
}
return f
}
func (f slotFrame) encode() string {
b := make([]byte, slotHeaderLen, slotHeaderLen+len(f.data))
copy(b, slotMagic)
binary.BigEndian.PutUint64(b[8:], uint64(f.seq))
b[16] = f.mode
binary.BigEndian.PutUint32(b[17:], uint32(f.size))
binary.BigEndian.PutUint32(b[21:], f.crc)
return base64.StdEncoding.EncodeToString(append(b, f.data...))
}
func decodeSlot(description string) (slotFrame, bool) {
b, err := base64.StdEncoding.DecodeString(description)
if err != nil || len(b) < slotHeaderLen || string(b[:len(slotMagic)]) != slotMagic {
return slotFrame{}, false
}
f := slotFrame{
seq: int64(binary.BigEndian.Uint64(b[8:])),
mode: b[16],
size: int(binary.BigEndian.Uint32(b[17:])),
crc: binary.BigEndian.Uint32(b[21:]),
}
if f.mode == slotInline {
f.data = b[slotHeaderLen:]
if !f.holds(f.data) {
return slotFrame{}, false
}
}
return f, true
}
// holds reports whether body is the one that f is about, which a body read
// right after its description need not be yet
func (f slotFrame) holds(body []byte) bool {
return len(body) == f.size && crc32.ChecksumIEEE(body) == f.crc
}
// slotTable is the ring of one direction: the ids of its slots and of the
// watermark of its reader
type slotTable struct {
Slots []string `json:"s"`
Watermark string `json:"w"`
}
// ring is what a client announces a session with, the tables are named from
// its side
type ring struct {
Up slotTable `json:"u"`
Down slotTable `json:"d"`
}
func decodeRing(b []byte) (ring, bool) {
var r ring
if json.Unmarshal(b, &r) != nil || len(r.Up.Slots) == 0 || len(r.Down.Slots) == 0 ||
r.Up.Watermark == "" || r.Down.Watermark == "" {
return ring{}, false
}
return r, true
}
// makeRing makes the slots and watermarks of a session, all at once
func makeRing(ctx context.Context, store slotStore, session string, p params) (ring, error) {
depth := min(max(p.concurrency, 2), slotDepthLimit)
r := ring{
Up: slotTable{Slots: make([]string, depth)},
Down: slotTable{Slots: make([]string, depth)},
}
var (
wg sync.WaitGroup
errOnce sync.Once
err error
)
create := func(id *string, name string) {
wg.Add(1)
go func() {
defer wg.Done()
var e error
if *id, e = store.makeSlot(ctx, name); e != nil {
errOnce.Do(func() { err = e })
}
}()
}
for _, t := range []struct {
table *slotTable
prefix string
}{{&r.Up, uplinkPrefix(session)}, {&r.Down, downlinkPrefix(session)}} {
for i := range t.table.Slots {
create(&t.table.Slots[i], t.prefix+"/s"+strconv.Itoa(i))
}
create(&t.table.Watermark, t.prefix+"/w")
}
wg.Wait()
return r, err
}
func readWatermark(ctx context.Context, store slotStore, id string) (int64, error) {
d, err := store.readDescription(ctx, id)
if err != nil || d == "" {
return -1, err
}
return strconv.ParseInt(d, 10, 64)
}
type slotWriter struct {
ctx context.Context
store slotStore
table slotTable
params
// slots has a token for each segment on its way, at most one for each slot
slots chan struct{}
wg sync.WaitGroup
mu sync.Mutex
buf []byte
seq int64
size int
full int
lastSize int
held int
closed bool
err error
// ackMu is held while the watermark is read, which other uploads wait for
ackMu sync.Mutex
acked int64
ackedAt time.Time
progress time.Time
}
func newSlotWriter(ctx context.Context, store slotStore, table slotTable, p params) *slotWriter {
w := &slotWriter{
ctx: ctx,
store: store,
table: table,
params: p,
slots: make(chan struct{}, len(table.Slots)),
size: min(slotInlineLimit, p.segmentBytes),
acked: -1,
progress: time.Now(),
}
go w.flushLoop()
return w
}
func (w *slotWriter) Write(p []byte) (int, error) {
w.mu.Lock()
defer w.mu.Unlock()
if w.err != nil {
return 0, w.err
}
if w.closed {
return 0, io.ErrClosedPipe
}
w.buf = append(w.buf, p...)
for len(w.buf) >= w.size {
if err := w.flushLocked(); err != nil {
return 0, err
}
}
return len(p), nil
}
func (w *slotWriter) flushLoop() {
ticker := time.NewTicker(w.flushInterval)
defer ticker.Stop()
for {
select {
case <-w.ctx.Done():
return
case <-ticker.C:
w.mu.Lock()
if !w.closed && w.err == nil && len(w.buf) > 0 && w.readyToFlush() {
w.flushLocked()
}
done := w.closed || w.err != nil
w.mu.Unlock()
if done {
return
}
}
}
}
func (w *slotWriter) readyToFlush() bool {
grew := len(w.buf) > w.lastSize
w.lastSize = len(w.buf)
if grew && w.held < maxCoalescedTicks {
w.held++
return false
}
w.held = 0
return true
}
// flushLocked sends a segment off buf once a slot is free, which it waits for
// without w.mu, so that a reader that falls behind holds Write back. w.mu is
// held
func (w *slotWriter) flushLocked() error {
w.mu.Unlock()
select {
case w.slots <- struct{}{}:
case <-w.ctx.Done():
w.mu.Lock()
return w.ctx.Err()
}
w.mu.Lock()
if w.err != nil || len(w.buf) == 0 {
<-w.slots
return w.err
}
n := min(len(w.buf), w.size)
if len(w.buf) > 4*slotInlineLimit {
// a burst goes at the full size at once
n = min(len(w.buf), w.segmentBytes)
}
if n == w.size && w.size < w.segmentBytes {
if w.full++; w.full == inlineRampSegments {
w.full = 0
w.size = min(2*w.size, w.segmentBytes)
}
}
chunk := bytes.Clone(w.buf[:n])
w.buf = append(w.buf[:0], w.buf[n:]...)
seq := w.seq
w.seq++
w.lastSize = len(w.buf)
w.held = 0
w.wg.Add(1)
go w.upload(seq, chunk)
return nil
}
func (w *slotWriter) slotID(seq int64) string {
return w.table.Slots[seq%int64(len(w.table.Slots))]
}
func (w *slotWriter) upload(seq int64, chunk []byte) {
defer w.wg.Done()
defer func() { <-w.slots }()
mode := byte(slotMedia)
if len(chunk) <= slotInlineLimit {
mode = slotInline
}
if err := w.await(seq); err != nil {
w.fail(errors.New("failed to store segment ", seq).Base(err))
return
}
if err := w.write(seq, mode, chunk); err != nil {
w.fail(errors.New("failed to store segment ", seq).Base(err))
// the slot is free, the reader learns there what happened
w.write(seq, slotErr, nil)
}
}
func (w *slotWriter) write(seq int64, mode byte, data []byte) error {
f := newSlotFrame(seq, mode, data)
var body []byte
if mode == slotMedia {
body = data
}
return w.store.setSlot(w.ctx, w.slotID(seq), f.encode(), body)
}
// drained reports whether the reader took what the slot of seq held a lap
// before. The watermark is read again only after slotWatermarkTTL, as one
// that is old costs a wait but never what is in a slot
func (w *slotWriter) drained(seq int64) (bool, error) {
w.ackMu.Lock()
defer w.ackMu.Unlock()
before := seq - int64(len(w.table.Slots))
if w.acked >= before {
return true, nil
}
if time.Since(w.ackedAt) < slotWatermarkTTL {
return false, nil
}
acked, err := readWatermark(w.ctx, w.store, w.table.Watermark)
if err != nil {
return false, err
}
w.ackedAt = time.Now()
if acked > w.acked {
w.acked, w.progress = acked, w.ackedAt
}
return w.acked >= before, nil
}
// await waits for drained, for slotStallTimeout at most, or slotCloseGrace
// without progress once the writer is closed
func (w *slotWriter) await(seq int64) error {
start := time.Now()
wait := slotAckRetry
for {
ok, err := w.drained(seq)
if ok {
return nil
}
w.mu.Lock()
closed := w.closed
w.mu.Unlock()
w.ackMu.Lock()
still := time.Since(w.progress)
w.ackMu.Unlock()
if time.Since(start) > slotStallTimeout || closed && still > slotCloseGrace {
return errors.New("the reader took nothing from slot ", w.slotID(seq), " for ", still).Base(err)
}
if err != nil {
wait = min(2*wait, 2*time.Second)
} else {
wait = slotAckRetry
}
select {
case <-w.ctx.Done():
return w.ctx.Err()
case <-time.After(wait):
}
}
}
func (w *slotWriter) fail(err error) {
w.mu.Lock()
defer w.mu.Unlock()
if w.err == nil {
w.err = err
}
}
func (w *slotWriter) Close() error {
w.mu.Lock()
if w.closed {
w.mu.Unlock()
return nil
}
w.closed = true
for len(w.buf) > 0 && w.err == nil {
if err := w.flushLocked(); err != nil {
w.mu.Unlock()
return err
}
}
seq := w.seq
w.mu.Unlock()
w.ackMu.Lock()
w.progress = time.Now()
w.ackMu.Unlock()
w.wg.Wait()
w.mu.Lock()
err := w.err
w.mu.Unlock()
if err != nil {
return err
}
if err := w.await(seq); err != nil {
return err
}
return w.write(seq, slotEnd, nil)
}
type slotReader struct {
ctx context.Context
store slotStore
table slotTable
params
ch chan []byte
wake chan struct{}
acks chan struct{}
mu sync.Mutex
head int64
started bool
// hitAt is when a segment last came and gap the time between them, activeAt
// when a segment came, was seen coming or Wake. Nothing for the eager window
// is idle, and active is closed when that ends
hitAt time.Time
gap time.Duration
activeAt time.Time
active chan struct{}
err error
}
func newSlotReader(ctx context.Context, store slotStore, table slotTable, p params) *slotReader {
r := &slotReader{
ctx: ctx,
store: store,
table: table,
params: p,
ch: make(chan []byte, p.concurrency),
wake: make(chan struct{}, 1),
acks: make(chan struct{}, 1),
hitAt: time.Now(),
activeAt: time.Now(),
active: make(chan struct{}),
}
go r.run()
go r.ackLoop()
return r
}
func (r *slotReader) chunks() <-chan []byte {
return r.ch
}
// Wake has the head asked for now, as an answer is likely to come
func (r *slotReader) Wake() {
r.mu.Lock()
r.touch()
r.mu.Unlock()
select {
case r.wake <- struct{}{}:
default:
}
}
func (r *slotReader) slotID(seq int64) string {
return r.table.Slots[seq%int64(len(r.table.Slots))]
}
type slotFetch struct {
ready chan struct{}
// promoted is closed when the segment becomes the head
promoted chan struct{}
body []byte
end bool
err error
}
func (r *slotReader) run() {
defer close(r.ch)
ctx, cancel := context.WithCancel(r.ctx)
defer cancel()
fetches := make(map[int64]*slotFetch)
for {
r.mu.Lock()
head, started := r.head, r.started
r.mu.Unlock()
window := 1
if started {
window = min(slotFetchAhead, len(r.table.Slots))
}
for i := range window {
seq := head + int64(i)
if fetches[seq] == nil {
fetches[seq] = &slotFetch{ready: make(chan struct{}), promoted: make(chan struct{})}
go r.fetch(ctx, seq, i, fetches[seq])
}
}
f := fetches[head]
select {
case <-f.ready:
case <-ctx.Done():
return
}
delete(fetches, head)
if f.err != nil {
r.setErr(f.err)
return
}
if f.end {
return
}
// the slot is free once the bytes are here, not once they are read
r.mu.Lock()
r.head++
r.started = true
r.mu.Unlock()
if next := fetches[head+1]; next != nil {
close(next.promoted)
}
select {
case r.acks <- struct{}{}:
default:
}
select {
case r.ch <- f.body:
case <-ctx.Done():
return
}
}
}
func (r *slotReader) isHead(seq int64) bool {
r.mu.Lock()
defer r.mu.Unlock()
return r.head == seq
}
// fetch asks for the slot of seq until it holds seq: the head often, a segment
// behind it at the pace that segments come, as it would wait for the head anyway
func (r *slotReader) fetch(ctx context.Context, seq int64, offset int, f *slotFetch) {
defer close(f.ready)
if offset > 0 && !r.hold(ctx, time.Duration(offset)*r.pace(), f.promoted, nil) {
return
}
id := r.slotID(seq)
backoff := slotHuntFloor
var goneSince, warnedAt time.Time
for {
description, err := r.store.readDescription(ctx, id)
frame, ok := decodeSlot(description)
switch {
case err == nil && ok && frame.seq == seq:
// what follows it is asked for while its body comes
r.mu.Lock()
r.touch()
r.mu.Unlock()
switch frame.mode {
case slotEnd:
f.end = true
return
case slotErr:
f.err = errors.New("the peer could not store segment ", seq)
return
case slotInline:
f.body = frame.data
r.hit()
return
case slotMedia:
if body, err := r.store.readBody(ctx, id); err == nil && frame.holds(body) {
f.body = body
r.hit()
return
}
}
case err == errNotFound:
if goneSince.IsZero() {
goneSince = time.Now()
} else if time.Since(goneSince) > slotGoneTimeout && r.isHead(seq) {
f.err = errors.New("slot ", id, " of segment ", seq, " is gone").Base(err)
return
}
case err != nil:
goneSince = time.Time{}
if ctx.Err() == nil && time.Since(warnedAt) > 10*time.Second {
warnedAt = time.Now()
errors.LogWarningInner(ctx, err, "failed to read slot ", id)
}
}
r.mu.Lock()
head, idle, active := r.head == seq, time.Since(r.activeAt) > r.eagerWindow, r.active
r.mu.Unlock()
wait, a, b := backoff, (<-chan struct{})(r.wake), (<-chan struct{})(nil)
switch {
case head:
ceiling := slotHuntCeiling
if idle {
ceiling = max(ceiling, r.maxPollInterval)
}
backoff = min(2*backoff, ceiling)
case idle:
// only the head is asked for until something comes
wait, a, b, backoff = 0, f.promoted, active, slotHuntFloor
default:
wait, a, backoff = min(max(r.pace(), slotTailMin), slotTailMax), f.promoted, slotHuntFloor
}
if !r.hold(ctx, wait, a, b) {
return
}
}
}
// pace is about how long segments take to come one after the other
func (r *slotReader) pace() time.Duration {
r.mu.Lock()
defer r.mu.Unlock()
return min(max(r.gap, 300*time.Millisecond), 3*time.Second)
}
func (r *slotReader) hit() {
r.mu.Lock()
defer r.mu.Unlock()
if gap := time.Since(r.hitAt); r.gap == 0 {
r.gap = gap
} else {
r.gap = (r.gap + gap) / 2
}
r.hitAt = time.Now()
r.touch()
}
// touch notes that something came or is likely to. r.mu is held
func (r *slotReader) touch() {
if time.Since(r.activeAt) > r.eagerWindow {
close(r.active)
r.active = make(chan struct{})
}
r.activeAt = time.Now()
}
// hold waits d, unless d is 0, or until a or b is closed or has a value
func (r *slotReader) hold(ctx context.Context, d time.Duration, a, b <-chan struct{}) bool {
var timeout <-chan time.Time
if d > 0 {
timer := time.NewTimer(d)
defer timer.Stop()
timeout = timer.C
}
select {
case <-ctx.Done():
return false
case <-a:
case <-b:
case <-timeout:
}
return true
}
// ackLoop writes the watermark when the head moves, the last head only if it
// moved more than once meanwhile
func (r *slotReader) ackLoop() {
acked := int64(-1)
wait := slotAckRetry
for {
select {
case <-r.ctx.Done():
return
case <-r.acks:
}
for {
r.mu.Lock()
want := r.head - 1
r.mu.Unlock()
if want <= acked {
break
}
if err := r.store.setSlot(r.ctx, r.table.Watermark, strconv.FormatInt(want, 10), nil); err == nil {
acked, wait = want, slotAckRetry
continue
}
wait = min(2*wait, 5*time.Second)
select {
case <-r.ctx.Done():
return
case <-time.After(wait):
}
}
}
}
func (r *slotReader) setErr(err error) {
r.mu.Lock()
defer r.mu.Unlock()
if r.err == nil {
r.err = err
}
}
func (r *slotReader) Err() error {
r.mu.Lock()
defer r.mu.Unlock()
if r.err != nil {
return r.err
}
return io.EOF
}
+309
View File
@@ -0,0 +1,309 @@
package xdrive
import (
"bytes"
"context"
"encoding/base64"
"errors"
"io"
"math/rand"
"os"
"strconv"
"sync"
"testing"
"time"
)
type memSlots struct {
mu sync.Mutex
next int
descs map[string]string
bodies map[string][]byte
reads int
}
func newMemSlots() *memSlots {
return &memSlots{descs: make(map[string]string), bodies: make(map[string][]byte)}
}
func (m *memSlots) makeSlot(ctx context.Context, name string) (string, error) {
m.mu.Lock()
defer m.mu.Unlock()
m.next++
id := "slot-" + strconv.Itoa(m.next)
m.descs[id] = ""
return id, nil
}
func (m *memSlots) setSlot(ctx context.Context, id, description string, body []byte) error {
m.mu.Lock()
defer m.mu.Unlock()
if _, ok := m.descs[id]; !ok {
return errNotFound
}
m.descs[id] = description
if body != nil {
m.bodies[id] = bytes.Clone(body)
}
return nil
}
func (m *memSlots) readDescription(ctx context.Context, id string) (string, error) {
m.mu.Lock()
defer m.mu.Unlock()
m.reads++
d, ok := m.descs[id]
if !ok {
return "", errNotFound
}
return d, nil
}
func (m *memSlots) readBody(ctx context.Context, id string) ([]byte, error) {
m.mu.Lock()
defer m.mu.Unlock()
return m.bodies[id], nil
}
// the slots of all tests are asked for often, set once as a goroutine of a
// closed Conn may still read them
func init() {
slotHuntFloor, slotHuntCeiling, slotTailMin, slotTailMax = 2*time.Millisecond, 10*time.Millisecond, 5*time.Millisecond, 20*time.Millisecond
slotAckRetry, slotWatermarkTTL = 2*time.Millisecond, 2*time.Millisecond
}
func slotPair(t *testing.T, store slotStore, concurrency, segmentBytes uint32) (*Conn, *Conn) {
t.Helper()
p := paramsFromConfig(&Config{
SegmentBytes: segmentBytes,
FlushIntervalMs: 1,
PollIntervalMs: 1,
MaxPollIntervalMs: 20,
EagerWindowMs: 50,
Concurrency: concurrency,
})
r, err := makeRing(context.Background(), store, "session", p)
if err != nil {
t.Fatal(err)
}
client := newSlotConn(context.Background(), store, r.Up, r.Down, p, nil)
server := newSlotConn(context.Background(), store, r.Down, r.Up, p, nil)
t.Cleanup(func() {
client.Close()
server.Close()
})
for _, c := range []*Conn{client, server} {
c.SetReadDeadline(time.Now().Add(30 * time.Second))
}
return client, server
}
// writeAll writes payload in pieces of random size, and closes c if end, the
// error goes to errs
func writeAll(c *Conn, payload []byte, end bool, errs chan<- error) {
go func() {
rng := rand.New(rand.NewSource(1))
for len(payload) > 0 {
n := min(len(payload), 1+rng.Intn(40<<10))
if _, err := c.Write(payload[:n]); err != nil {
errs <- err
return
}
payload = payload[n:]
}
var err error
if end {
err = c.Close()
}
errs <- err
}()
}
func readAll(t *testing.T, c *Conn, want []byte) {
t.Helper()
got, err := io.ReadAll(c)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, want) {
t.Fatalf("got %d bytes, not the %d that were sent", len(got), len(want))
}
}
func randomPayload(n int) []byte {
payload := make([]byte, n)
rand.New(rand.NewSource(int64(n))).Read(payload)
return payload
}
func TestSlotFrame(t *testing.T) {
data := []byte("hello")
f, ok := decodeSlot(newSlotFrame(7, slotInline, data).encode())
if !ok || f.seq != 7 || f.mode != slotInline || !bytes.Equal(f.data, data) {
t.Fatalf("inline frame came back as %+v %v", f, ok)
}
body := randomPayload(100)
f, ok = decodeSlot(newSlotFrame(8, slotMedia, body).encode())
if !ok || f.seq != 8 || f.mode != slotMedia || f.data != nil || !f.holds(body) || f.holds(body[1:]) {
t.Fatalf("media frame came back as %+v %v", f, ok)
}
corrupt, _ := base64.StdEncoding.DecodeString(newSlotFrame(1, slotInline, data).encode())
corrupt[len(corrupt)-1]++
for _, broken := range []string{
"", "not base64", newSlotFrame(1, slotInline, data).encode()[:20], "WERSVlNMMDE=",
base64.StdEncoding.EncodeToString(corrupt),
} {
if _, ok := decodeSlot(broken); ok {
t.Errorf("%q decoded", broken)
}
}
}
// both sides send more segments than the ring has slots at once, small ones
// that go in descriptions and large ones that go in bodies, then one closes
// and the other reads EOF
func TestSlotConnRoundTrip(t *testing.T) {
client, server := slotPair(t, newMemSlots(), 4, 64<<10)
up, down := randomPayload(1<<20), randomPayload(300<<10)
errs := make(chan error, 2)
writeAll(client, up, false, errs)
writeAll(server, down, false, errs)
for _, c := range []struct {
conn *Conn
want []byte
}{{server, up}, {client, down}} {
got := make([]byte, len(c.want))
if _, err := io.ReadFull(c.conn, got); err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, c.want) {
t.Fatal("what came is not what was sent")
}
}
for range 2 {
if err := <-errs; err != nil {
t.Fatal(err)
}
}
if err := client.Close(); err != nil {
t.Fatal(err)
}
readAll(t, server, nil)
}
// a reader that does not read holds the writer back instead of losing what a
// lap of the ring would overwrite
func TestSlotConnSlowReader(t *testing.T) {
client, server := slotPair(t, newMemSlots(), 2, 16<<10)
payload := randomPayload(400 << 10)
errs := make(chan error, 1)
writeAll(client, payload, true, errs)
time.Sleep(200 * time.Millisecond)
readAll(t, server, payload)
if err := <-errs; err != nil {
t.Fatal(err)
}
}
// the next segment is asked for at once when it becomes the head, not after
// the wait it had as one that was fetched ahead
func TestSlotLatency(t *testing.T) {
client, server := slotPair(t, newMemSlots(), 8, 64<<10)
start := time.Now()
for range 10 {
for _, c := range [][2]*Conn{{client, server}, {server, client}} {
c[0].Write([]byte("ping"))
if _, err := io.ReadFull(c[1], make([]byte, 4)); err != nil {
t.Fatal(err)
}
}
}
if took := time.Since(start); took > 2*time.Second {
t.Fatalf("10 round trips took %v", took)
}
}
// once nothing comes for the eager window only the heads are asked for, not
// the segments that a reader fetches ahead while things come
func TestSlotIdle(t *testing.T) {
store := newMemSlots()
client, server := slotPair(t, store, 8, 64<<10)
for _, c := range [][2]*Conn{{client, server}, {server, client}} {
c[0].Write([]byte("ping"))
if _, err := io.ReadFull(c[1], make([]byte, 4)); err != nil {
t.Fatal(err)
}
}
time.Sleep(200 * time.Millisecond)
store.mu.Lock()
before := store.reads
store.mu.Unlock()
time.Sleep(500 * time.Millisecond)
store.mu.Lock()
reads := store.reads - before
store.mu.Unlock()
// two heads, each asked for every 20 ms at most
if reads > 75 {
t.Fatalf("an idle session asked for slots %d times in 500 ms", reads)
}
}
type failingSlots struct {
*memSlots
}
// setSlot fails for segments, not for the error frame
func (f failingSlots) setSlot(ctx context.Context, id, description string, body []byte) error {
if frame, ok := decodeSlot(description); ok && (frame.mode == slotInline || frame.mode == slotMedia) {
return io.ErrUnexpectedEOF
}
return f.memSlots.setSlot(ctx, id, description, body)
}
func TestSlotWriterFailure(t *testing.T) {
client, server := slotPair(t, failingSlots{newMemSlots()}, 2, 16<<10)
client.Write([]byte("lost"))
server.SetReadDeadline(time.Now().Add(5 * time.Second))
if _, err := server.Read(make([]byte, 10)); err == nil || err == io.EOF || errors.Is(err, os.ErrDeadlineExceeded) {
t.Fatalf("the reader got %v, not the failure of the writer", err)
}
for start := time.Now(); time.Since(start) < 5*time.Second; time.Sleep(time.Millisecond) {
if _, err := client.Write([]byte("more")); err != nil {
return
}
}
t.Fatal("Write did not fail")
}
type staleSlots struct {
*memSlots
mu sync.Mutex
served map[string]bool
}
// readBody gives what a slot held before once, as a body read right after its
// description may
func (s *staleSlots) readBody(ctx context.Context, id string) ([]byte, error) {
s.mu.Lock()
first := !s.served[id]
s.served[id] = true
s.mu.Unlock()
if first {
return []byte("stale"), nil
}
return s.memSlots.readBody(ctx, id)
}
func TestSlotStaleBody(t *testing.T) {
client, server := slotPair(t, &staleSlots{memSlots: newMemSlots(), served: make(map[string]bool)}, 2, 64<<10)
payload := randomPayload(200 << 10)
errs := make(chan error, 1)
writeAll(client, payload, true, errs)
readAll(t, server, payload)
}
+14
View File
@@ -43,6 +43,10 @@ func newStorage(streamSettings *internet.MemoryStreamConfig) (Storage, error) {
return sharedStorage(streamSettings, config, func() (Storage, error) {
return newTemplateStorage(streamSettings, config)
})
case "s3":
return sharedStorage(streamSettings, config, func() (Storage, error) {
return newS3Storage(streamSettings, config)
})
default:
return nil, errors.New("unsupported service: ", config.Service)
}
@@ -57,6 +61,8 @@ func sessionStorage(storage Storage) Storage {
return s.session()
case *templateStorage:
return s.session()
case *s3Storage:
return s.session()
}
return storage
}
@@ -68,6 +74,14 @@ var (
func shareKey(streamSettings *internet.MemoryStreamConfig, config *Config) string {
parts := []string{config.Service, config.RemoteFolder}
if config.S3 != nil {
parts = append(parts, config.S3.Endpoint, s3Region(config.S3.Region), config.S3.Bucket)
if config.S3.PathStyle {
parts = append(parts, "path-style")
} else {
parts = append(parts, "virtual-hosted-style")
}
}
parts = append(parts, config.Secrets...)
if streamSettings != nil {
parts = append(parts, streamSettings.SecurityType)
+4
View File
@@ -230,6 +230,10 @@ func newWALReader(ctx context.Context, storage Storage, prefix string, p params)
return r
}
func (r *walReader) chunks() <-chan []byte {
return r.ch
}
func (r *walReader) Wake() {
select {
case r.wake <- struct{}{}:
+40 -13
View File
@@ -4,6 +4,7 @@ import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"fmt"
"strconv"
"strings"
@@ -95,15 +96,33 @@ func Dial(ctx context.Context, dest net.Destination, streamSettings *internet.Me
return nil, err
}
if err := storage.Put(ctx, announceName(session, time.Now()), nil); err != nil {
p := paramsFromConfig(config)
var announce []byte
var r ring
slots, ok := storage.(slotStore)
if ok {
if r, err = makeRing(ctx, slots, session, p); err == nil {
announce, _ = json.Marshal(r)
} else {
ok = false
errors.LogWarningInner(ctx, err, "failed to make the slots of session ", session, ", it goes in segments")
}
}
if err := storage.Put(ctx, announceName(session, time.Now()), announce); err != nil {
storage.Close()
return nil, errors.New("failed to announce session ", session).Base(err)
}
if ok {
errors.LogInfo(ctx, "opened session ", session, " in ", len(r.Up.Slots), " slots")
return newSlotConn(context.Background(), slots, r.Up, r.Down, p, func() {
storage.Close()
}), nil
}
errors.LogInfo(ctx, "opened session ", session)
return newConn(context.Background(), storage,
uplinkPrefix(session), downlinkPrefix(session), paramsFromConfig(config), func() {
uplinkPrefix(session), downlinkPrefix(session), p, func() {
storage.Close()
}), nil
}
@@ -206,9 +225,8 @@ func (l *Listener) acceptPending(logCtx context.Context) (bool, error) {
continue
}
go l.drop(full)
errors.LogInfo(logCtx, "accepted session ", session)
accepted = true
l.addConn(l.newSessionConn(session))
l.addConn(l.newSessionConn(logCtx, session, listed.Inline))
}
return accepted, nil
}
@@ -228,15 +246,24 @@ func (l *Listener) claim(session string) bool {
return true
}
func (l *Listener) newSessionConn(session string) *Conn {
// newSessionConn takes a session in the slots that its announcement has, or
// in segments if it has none
func (l *Listener) newSessionConn(logCtx context.Context, session string, announce []byte) *Conn {
storage := sessionStorage(l.storage)
return newConn(l.ctx, storage,
downlinkPrefix(session), uplinkPrefix(session), l.params, func() {
storage.Close()
l.mu.Lock()
delete(l.active, session)
l.mu.Unlock()
})
onClose := func() {
storage.Close()
l.mu.Lock()
delete(l.active, session)
l.mu.Unlock()
}
if slots, ok := storage.(slotStore); ok {
if r, ok := decodeRing(announce); ok {
errors.LogInfo(logCtx, "accepted session ", session, " in ", len(r.Down.Slots), " slots")
return newSlotConn(l.ctx, slots, r.Down, r.Up, l.params, onClose)
}
}
errors.LogInfo(logCtx, "accepted session ", session)
return newConn(l.ctx, storage, downlinkPrefix(session), uplinkPrefix(session), l.params, onClose)
}
func (l *Listener) collectLoop(logCtx context.Context) {