XiaoMo/frp
1
0
Fork 0
mirror of https://github.com/fatedier/frp.git synced 2026-01-11 22:23:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

use dep instead of glide

Browse Source
This commit is contained in:
fatedier
2018-05-08 02:35:13 +08:00
parent 8a6d6c534a
commit dd8f788ca4
1209 changed files with 7782 additions and 310762 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
vendor/github.com/klauspost/cpuid/.gitignore generated vendored
View File

@@ -1,24 +0,0 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
*.test
*.prof

8
vendor/github.com/klauspost/cpuid/.travis.yml generated vendored
View File

@@ -1,8 +0,0 @@
language: go
go:
- 1.3
- 1.4
- 1.5
- 1.6
- tip

22
vendor/github.com/klauspost/cpuid/LICENSE generated vendored
View File

@@ -1,22 +0,0 @@
The MIT License (MIT)
Copyright (c) 2015 Klaus Post
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

145
vendor/github.com/klauspost/cpuid/README.md generated vendored
View File

@@ -1,145 +0,0 @@
# cpuid
Package cpuid provides information about the CPU running the current program.
CPU features are detected on startup, and kept for fast access through the life of the application.
Currently x86 / x64 (AMD64) is supported, and no external C (cgo) code is used, which should make the library very easy to use.
You can access the CPU information by accessing the shared CPU variable of the cpuid library.
Package home: https://github.com/klauspost/cpuid
[![GoDoc][1]][2] [![Build Status][3]][4]
[1]: https://godoc.org/github.com/klauspost/cpuid?status.svg
[2]: https://godoc.org/github.com/klauspost/cpuid
[3]: https://travis-ci.org/klauspost/cpuid.svg
[4]: https://travis-ci.org/klauspost/cpuid
# features
## CPU Instructions
* **CMOV** (i686 CMOV)
* **NX** (NX (No-Execute) bit)
* **AMD3DNOW** (AMD 3DNOW)
* **AMD3DNOWEXT** (AMD 3DNowExt)
* **MMX** (standard MMX)
* **MMXEXT** (SSE integer functions or AMD MMX ext)
* **SSE** (SSE functions)
* **SSE2** (P4 SSE functions)
* **SSE3** (Prescott SSE3 functions)
* **SSSE3** (Conroe SSSE3 functions)
* **SSE4** (Penryn SSE4.1 functions)
* **SSE4A** (AMD Barcelona microarchitecture SSE4a instructions)
* **SSE42** (Nehalem SSE4.2 functions)
* **AVX** (AVX functions)
* **AVX2** (AVX2 functions)
* **FMA3** (Intel FMA 3)
* **FMA4** (Bulldozer FMA4 functions)
* **XOP** (Bulldozer XOP functions)
* **F16C** (Half-precision floating-point conversion)
* **BMI1** (Bit Manipulation Instruction Set 1)
* **BMI2** (Bit Manipulation Instruction Set 2)
* **TBM** (AMD Trailing Bit Manipulation)
* **LZCNT** (LZCNT instruction)
* **POPCNT** (POPCNT instruction)
* **AESNI** (Advanced Encryption Standard New Instructions)
* **CLMUL** (Carry-less Multiplication)
* **HTT** (Hyperthreading (enabled))
* **HLE** (Hardware Lock Elision)
* **RTM** (Restricted Transactional Memory)
* **RDRAND** (RDRAND instruction is available)
* **RDSEED** (RDSEED instruction is available)
* **ADX** (Intel ADX (Multi-Precision Add-Carry Instruction Extensions))
* **SHA** (Intel SHA Extensions)
* **AVX512F** (AVX-512 Foundation)
* **AVX512DQ** (AVX-512 Doubleword and Quadword Instructions)
* **AVX512IFMA** (AVX-512 Integer Fused Multiply-Add Instructions)
* **AVX512PF** (AVX-512 Prefetch Instructions)
* **AVX512ER** (AVX-512 Exponential and Reciprocal Instructions)
* **AVX512CD** (AVX-512 Conflict Detection Instructions)
* **AVX512BW** (AVX-512 Byte and Word Instructions)
* **AVX512VL** (AVX-512 Vector Length Extensions)
* **AVX512VBMI** (AVX-512 Vector Bit Manipulation Instructions)
* **MPX** (Intel MPX (Memory Protection Extensions))
* **ERMS** (Enhanced REP MOVSB/STOSB)
* **RDTSCP** (RDTSCP Instruction)
* **CX16** (CMPXCHG16B Instruction)
* **SGX** (Software Guard Extensions, with activation details)
## Performance
* **RDTSCP()** Returns current cycle count. Can be used for benchmarking.
* **SSE2SLOW** (SSE2 is supported, but usually not faster)
* **SSE3SLOW** (SSE3 is supported, but usually not faster)
* **ATOM** (Atom processor, some SSSE3 instructions are slower)
* **Cache line** (Probable size of a cache line).
* **L1, L2, L3 Cache size** on newer Intel/AMD CPUs.
## Cpu Vendor/VM
* **Intel**
* **AMD**
* **VIA**
* **Transmeta**
* **NSC**
* **KVM** (Kernel-based Virtual Machine)
* **MSVM** (Microsoft Hyper-V or Windows Virtual PC)
* **VMware**
* **XenHVM**
# installing
```go get github.com/klauspost/cpuid```
# example
```Go
package main
import (
"fmt"
"github.com/klauspost/cpuid"
)
func main() {
// Print basic CPU information:
fmt.Println("Name:", cpuid.CPU.BrandName)
fmt.Println("PhysicalCores:", cpuid.CPU.PhysicalCores)
fmt.Println("ThreadsPerCore:", cpuid.CPU.ThreadsPerCore)
fmt.Println("LogicalCores:", cpuid.CPU.LogicalCores)
fmt.Println("Family", cpuid.CPU.Family, "Model:", cpuid.CPU.Model)
fmt.Println("Features:", cpuid.CPU.Features)
fmt.Println("Cacheline bytes:", cpuid.CPU.CacheLine)
fmt.Println("L1 Data Cache:", cpuid.CPU.Cache.L1D, "bytes")
fmt.Println("L1 Instruction Cache:", cpuid.CPU.Cache.L1D, "bytes")
fmt.Println("L2 Cache:", cpuid.CPU.Cache.L2, "bytes")
fmt.Println("L3 Cache:", cpuid.CPU.Cache.L3, "bytes")
// Test if we have a specific feature:
if cpuid.CPU.SSE() {
fmt.Println("We have Streaming SIMD Extensions")
}
}
```
Sample output:
```
>go run main.go
Name: Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz
PhysicalCores: 2
ThreadsPerCore: 2
LogicalCores: 4
Family 6 Model: 42
Features: CMOV,MMX,MMXEXT,SSE,SSE2,SSE3,SSSE3,SSE4.1,SSE4.2,AVX,AESNI,CLMUL
Cacheline bytes: 64
We have Streaming SIMD Extensions
```
# private package
In the "private" folder you can find an autogenerated version of the library you can include in your own packages.
For this purpose all exports are removed, and functions and constants are lowercased.
This is not a recommended way of using the library, but provided for convenience, if it is difficult for you to use external packages.
# license
This code is published under an MIT license. See LICENSE file for more information.

1022
vendor/github.com/klauspost/cpuid/cpuid.go generated vendored
View File

File diff suppressed because it is too large Load Diff

42
vendor/github.com/klauspost/cpuid/cpuid_386.s generated vendored
View File

@@ -1,42 +0,0 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORL CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+4(FP)
MOVL BX, ebx+8(FP)
MOVL CX, ecx+12(FP)
MOVL DX, edx+16(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func xgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+4(FP)
MOVL DX, edx+8(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET

42
vendor/github.com/klauspost/cpuid/cpuid_amd64.s generated vendored
View File

@@ -1,42 +0,0 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
//+build amd64,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORQ CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmXgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+8(FP)
MOVL DX, edx+12(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET

727
vendor/github.com/klauspost/cpuid/cpuid_test.go generated vendored
View File

@@ -1,727 +0,0 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
package cpuid
import (
"fmt"
"testing"
)
// There is no real way to test a CPU identifier, since results will
// obviously differ on each machine.
func TestCPUID(t *testing.T) {
n := maxFunctionID()
t.Logf("Max Function:0x%x\n", n)
n = maxExtendedFunction()
t.Logf("Max Extended Function:0x%x\n", n)
t.Log("Name:", CPU.BrandName)
t.Log("PhysicalCores:", CPU.PhysicalCores)
t.Log("ThreadsPerCore:", CPU.ThreadsPerCore)
t.Log("LogicalCores:", CPU.LogicalCores)
t.Log("Family", CPU.Family, "Model:", CPU.Model)
t.Log("Features:", CPU.Features)
t.Log("Cacheline bytes:", CPU.CacheLine)
t.Log("L1 Instruction Cache:", CPU.Cache.L1I, "bytes")
t.Log("L1 Data Cache:", CPU.Cache.L1D, "bytes")
t.Log("L2 Cache:", CPU.Cache.L2, "bytes")
t.Log("L3 Cache:", CPU.Cache.L3, "bytes")
if CPU.SSE2() {
t.Log("We have SSE2")
}
}
func TestDumpCPUID(t *testing.T) {
n := int(maxFunctionID())
for i := 0; i <= n; i++ {
a, b, c, d := cpuidex(uint32(i), 0)
t.Logf("CPUID %08x: %08x-%08x-%08x-%08x", i, a, b, c, d)
ex := uint32(1)
for {
a2, b2, c2, d2 := cpuidex(uint32(i), ex)
if a2 == a && b2 == b && d2 == d || ex > 50 || a2 == 0 {
break
}
t.Logf("CPUID %08x: %08x-%08x-%08x-%08x", i, a2, b2, c2, d2)
a, b, c, d = a2, b2, c2, d2
ex++
}
}
n2 := maxExtendedFunction()
for i := uint32(0x80000000); i <= n2; i++ {
a, b, c, d := cpuid(i)
t.Logf("CPUID %08x: %08x-%08x-%08x-%08x", i, a, b, c, d)
}
}
func Example() {
// Print basic CPU information:
fmt.Println("Name:", CPU.BrandName)
fmt.Println("PhysicalCores:", CPU.PhysicalCores)
fmt.Println("ThreadsPerCore:", CPU.ThreadsPerCore)
fmt.Println("LogicalCores:", CPU.LogicalCores)
fmt.Println("Family", CPU.Family, "Model:", CPU.Model)
fmt.Println("Features:", CPU.Features)
fmt.Println("Cacheline bytes:", CPU.CacheLine)
// Test if we have a specific feature:
if CPU.SSE() {
fmt.Println("We have Streaming SIMD Extensions")
}
}
func TestBrandNameZero(t *testing.T) {
if len(CPU.BrandName) > 0 {
// Cut out last byte
last := []byte(CPU.BrandName[len(CPU.BrandName)-1:])
if last[0] == 0 {
t.Fatal("last byte was zero")
} else if last[0] == 32 {
t.Fatal("whitespace wasn't trimmed")
}
}
}
// Generated here: http://play.golang.org/p/mko-0tFt0Q
// TestCmov tests Cmov() function
func TestCmov(t *testing.T) {
got := CPU.Cmov()
expected := CPU.Features&CMOV == CMOV
if got != expected {
t.Fatalf("Cmov: expected %v, got %v", expected, got)
}
t.Log("CMOV Support:", got)
}
// TestAmd3dnow tests Amd3dnow() function
func TestAmd3dnow(t *testing.T) {
got := CPU.Amd3dnow()
expected := CPU.Features&AMD3DNOW == AMD3DNOW
if got != expected {
t.Fatalf("Amd3dnow: expected %v, got %v", expected, got)
}
t.Log("AMD3DNOW Support:", got)
}
// TestAmd3dnowExt tests Amd3dnowExt() function
func TestAmd3dnowExt(t *testing.T) {
got := CPU.Amd3dnowExt()
expected := CPU.Features&AMD3DNOWEXT == AMD3DNOWEXT
if got != expected {
t.Fatalf("Amd3dnowExt: expected %v, got %v", expected, got)
}
t.Log("AMD3DNOWEXT Support:", got)
}
// TestMMX tests MMX() function
func TestMMX(t *testing.T) {
got := CPU.MMX()
expected := CPU.Features&MMX == MMX
if got != expected {
t.Fatalf("MMX: expected %v, got %v", expected, got)
}
t.Log("MMX Support:", got)
}
// TestMMXext tests MMXext() function
func TestMMXext(t *testing.T) {
got := CPU.MMXExt()
expected := CPU.Features&MMXEXT == MMXEXT
if got != expected {
t.Fatalf("MMXExt: expected %v, got %v", expected, got)
}
t.Log("MMXEXT Support:", got)
}
// TestSSE tests SSE() function
func TestSSE(t *testing.T) {
got := CPU.SSE()
expected := CPU.Features&SSE == SSE
if got != expected {
t.Fatalf("SSE: expected %v, got %v", expected, got)
}
t.Log("SSE Support:", got)
}
// TestSSE2 tests SSE2() function
func TestSSE2(t *testing.T) {
got := CPU.SSE2()
expected := CPU.Features&SSE2 == SSE2
if got != expected {
t.Fatalf("SSE2: expected %v, got %v", expected, got)
}
t.Log("SSE2 Support:", got)
}
// TestSSE3 tests SSE3() function
func TestSSE3(t *testing.T) {
got := CPU.SSE3()
expected := CPU.Features&SSE3 == SSE3
if got != expected {
t.Fatalf("SSE3: expected %v, got %v", expected, got)
}
t.Log("SSE3 Support:", got)
}
// TestSSSE3 tests SSSE3() function
func TestSSSE3(t *testing.T) {
got := CPU.SSSE3()
expected := CPU.Features&SSSE3 == SSSE3
if got != expected {
t.Fatalf("SSSE3: expected %v, got %v", expected, got)
}
t.Log("SSSE3 Support:", got)
}
// TestSSE4 tests SSE4() function
func TestSSE4(t *testing.T) {
got := CPU.SSE4()
expected := CPU.Features&SSE4 == SSE4
if got != expected {
t.Fatalf("SSE4: expected %v, got %v", expected, got)
}
t.Log("SSE4 Support:", got)
}
// TestSSE42 tests SSE42() function
func TestSSE42(t *testing.T) {
got := CPU.SSE42()
expected := CPU.Features&SSE42 == SSE42
if got != expected {
t.Fatalf("SSE42: expected %v, got %v", expected, got)
}
t.Log("SSE42 Support:", got)
}
// TestAVX tests AVX() function
func TestAVX(t *testing.T) {
got := CPU.AVX()
expected := CPU.Features&AVX == AVX
if got != expected {
t.Fatalf("AVX: expected %v, got %v", expected, got)
}
t.Log("AVX Support:", got)
}
// TestAVX2 tests AVX2() function
func TestAVX2(t *testing.T) {
got := CPU.AVX2()
expected := CPU.Features&AVX2 == AVX2
if got != expected {
t.Fatalf("AVX2: expected %v, got %v", expected, got)
}
t.Log("AVX2 Support:", got)
}
// TestFMA3 tests FMA3() function
func TestFMA3(t *testing.T) {
got := CPU.FMA3()
expected := CPU.Features&FMA3 == FMA3
if got != expected {
t.Fatalf("FMA3: expected %v, got %v", expected, got)
}
t.Log("FMA3 Support:", got)
}
// TestFMA4 tests FMA4() function
func TestFMA4(t *testing.T) {
got := CPU.FMA4()
expected := CPU.Features&FMA4 == FMA4
if got != expected {
t.Fatalf("FMA4: expected %v, got %v", expected, got)
}
t.Log("FMA4 Support:", got)
}
// TestXOP tests XOP() function
func TestXOP(t *testing.T) {
got := CPU.XOP()
expected := CPU.Features&XOP == XOP
if got != expected {
t.Fatalf("XOP: expected %v, got %v", expected, got)
}
t.Log("XOP Support:", got)
}
// TestF16C tests F16C() function
func TestF16C(t *testing.T) {
got := CPU.F16C()
expected := CPU.Features&F16C == F16C
if got != expected {
t.Fatalf("F16C: expected %v, got %v", expected, got)
}
t.Log("F16C Support:", got)
}
// TestCX16 tests CX16() function
func TestCX16(t *testing.T) {
got := CPU.CX16()
expected := CPU.Features&CX16 == CX16
if got != expected {
t.Fatalf("CX16: expected %v, got %v", expected, got)
}
t.Log("CX16 Support:", got)
}
// TestSGX tests SGX() function
func TestSGX(t *testing.T) {
got := CPU.SGX.Available
expected := CPU.Features&SGX == SGX
if got != expected {
t.Fatalf("SGX: expected %v, got %v", expected, got)
}
t.Log("SGX Support:", got)
}
// TestBMI1 tests BMI1() function
func TestBMI1(t *testing.T) {
got := CPU.BMI1()
expected := CPU.Features&BMI1 == BMI1
if got != expected {
t.Fatalf("BMI1: expected %v, got %v", expected, got)
}
t.Log("BMI1 Support:", got)
}
// TestBMI2 tests BMI2() function
func TestBMI2(t *testing.T) {
got := CPU.BMI2()
expected := CPU.Features&BMI2 == BMI2
if got != expected {
t.Fatalf("BMI2: expected %v, got %v", expected, got)
}
t.Log("BMI2 Support:", got)
}
// TestTBM tests TBM() function
func TestTBM(t *testing.T) {
got := CPU.TBM()
expected := CPU.Features&TBM == TBM
if got != expected {
t.Fatalf("TBM: expected %v, got %v", expected, got)
}
t.Log("TBM Support:", got)
}
// TestLzcnt tests Lzcnt() function
func TestLzcnt(t *testing.T) {
got := CPU.Lzcnt()
expected := CPU.Features&LZCNT == LZCNT
if got != expected {
t.Fatalf("Lzcnt: expected %v, got %v", expected, got)
}
t.Log("LZCNT Support:", got)
}
// TestLzcnt tests Lzcnt() function
func TestPopcnt(t *testing.T) {
got := CPU.Popcnt()
expected := CPU.Features&POPCNT == POPCNT
if got != expected {
t.Fatalf("Popcnt: expected %v, got %v", expected, got)
}
t.Log("POPCNT Support:", got)
}
// TestAesNi tests AesNi() function
func TestAesNi(t *testing.T) {
got := CPU.AesNi()
expected := CPU.Features&AESNI == AESNI
if got != expected {
t.Fatalf("AesNi: expected %v, got %v", expected, got)
}
t.Log("AESNI Support:", got)
}
// TestHTT tests HTT() function
func TestHTT(t *testing.T) {
got := CPU.HTT()
expected := CPU.Features&HTT == HTT
if got != expected {
t.Fatalf("HTT: expected %v, got %v", expected, got)
}
t.Log("HTT Support:", got)
}
// TestClmul tests Clmul() function
func TestClmul(t *testing.T) {
got := CPU.Clmul()
expected := CPU.Features&CLMUL == CLMUL
if got != expected {
t.Fatalf("Clmul: expected %v, got %v", expected, got)
}
t.Log("CLMUL Support:", got)
}
// TestSSE2Slow tests SSE2Slow() function
func TestSSE2Slow(t *testing.T) {
got := CPU.SSE2Slow()
expected := CPU.Features&SSE2SLOW == SSE2SLOW
if got != expected {
t.Fatalf("SSE2Slow: expected %v, got %v", expected, got)
}
t.Log("SSE2SLOW Support:", got)
}
// TestSSE3Slow tests SSE3slow() function
func TestSSE3Slow(t *testing.T) {
got := CPU.SSE3Slow()
expected := CPU.Features&SSE3SLOW == SSE3SLOW
if got != expected {
t.Fatalf("SSE3slow: expected %v, got %v", expected, got)
}
t.Log("SSE3SLOW Support:", got)
}
// TestAtom tests Atom() function
func TestAtom(t *testing.T) {
got := CPU.Atom()
expected := CPU.Features&ATOM == ATOM
if got != expected {
t.Fatalf("Atom: expected %v, got %v", expected, got)
}
t.Log("ATOM Support:", got)
}
// TestNX tests NX() function (NX (No-Execute) bit)
func TestNX(t *testing.T) {
got := CPU.NX()
expected := CPU.Features&NX == NX
if got != expected {
t.Fatalf("NX: expected %v, got %v", expected, got)
}
t.Log("NX Support:", got)
}
// TestSSE4A tests SSE4A() function (AMD Barcelona microarchitecture SSE4a instructions)
func TestSSE4A(t *testing.T) {
got := CPU.SSE4A()
expected := CPU.Features&SSE4A == SSE4A
if got != expected {
t.Fatalf("SSE4A: expected %v, got %v", expected, got)
}
t.Log("SSE4A Support:", got)
}
// TestHLE tests HLE() function (Hardware Lock Elision)
func TestHLE(t *testing.T) {
got := CPU.HLE()
expected := CPU.Features&HLE == HLE
if got != expected {
t.Fatalf("HLE: expected %v, got %v", expected, got)
}
t.Log("HLE Support:", got)
}
// TestRTM tests RTM() function (Restricted Transactional Memory)
func TestRTM(t *testing.T) {
got := CPU.RTM()
expected := CPU.Features&RTM == RTM
if got != expected {
t.Fatalf("RTM: expected %v, got %v", expected, got)
}
t.Log("RTM Support:", got)
}
// TestRdrand tests RDRAND() function (RDRAND instruction is available)
func TestRdrand(t *testing.T) {
got := CPU.Rdrand()
expected := CPU.Features&RDRAND == RDRAND
if got != expected {
t.Fatalf("Rdrand: expected %v, got %v", expected, got)
}
t.Log("Rdrand Support:", got)
}
// TestRdseed tests RDSEED() function (RDSEED instruction is available)
func TestRdseed(t *testing.T) {
got := CPU.Rdseed()
expected := CPU.Features&RDSEED == RDSEED
if got != expected {
t.Fatalf("Rdseed: expected %v, got %v", expected, got)
}
t.Log("Rdseed Support:", got)
}
// TestADX tests ADX() function (Intel ADX (Multi-Precision Add-Carry Instruction Extensions))
func TestADX(t *testing.T) {
got := CPU.ADX()
expected := CPU.Features&ADX == ADX
if got != expected {
t.Fatalf("ADX: expected %v, got %v", expected, got)
}
t.Log("ADX Support:", got)
}
// TestSHA tests SHA() function (Intel SHA Extensions)
func TestSHA(t *testing.T) {
got := CPU.SHA()
expected := CPU.Features&SHA == SHA
if got != expected {
t.Fatalf("SHA: expected %v, got %v", expected, got)
}
t.Log("SHA Support:", got)
}
// TestAVX512F tests AVX512F() function (AVX-512 Foundation)
func TestAVX512F(t *testing.T) {
got := CPU.AVX512F()
expected := CPU.Features&AVX512F == AVX512F
if got != expected {
t.Fatalf("AVX512F: expected %v, got %v", expected, got)
}
t.Log("AVX512F Support:", got)
}
// TestAVX512DQ tests AVX512DQ() function (AVX-512 Doubleword and Quadword Instructions)
func TestAVX512DQ(t *testing.T) {
got := CPU.AVX512DQ()
expected := CPU.Features&AVX512DQ == AVX512DQ
if got != expected {
t.Fatalf("AVX512DQ: expected %v, got %v", expected, got)
}
t.Log("AVX512DQ Support:", got)
}
// TestAVX512IFMA tests AVX512IFMA() function (AVX-512 Integer Fused Multiply-Add Instructions)
func TestAVX512IFMA(t *testing.T) {
got := CPU.AVX512IFMA()
expected := CPU.Features&AVX512IFMA == AVX512IFMA
if got != expected {
t.Fatalf("AVX512IFMA: expected %v, got %v", expected, got)
}
t.Log("AVX512IFMA Support:", got)
}
// TestAVX512PF tests AVX512PF() function (AVX-512 Prefetch Instructions)
func TestAVX512PF(t *testing.T) {
got := CPU.AVX512PF()
expected := CPU.Features&AVX512PF == AVX512PF
if got != expected {
t.Fatalf("AVX512PF: expected %v, got %v", expected, got)
}
t.Log("AVX512PF Support:", got)
}
// TestAVX512ER tests AVX512ER() function (AVX-512 Exponential and Reciprocal Instructions)
func TestAVX512ER(t *testing.T) {
got := CPU.AVX512ER()
expected := CPU.Features&AVX512ER == AVX512ER
if got != expected {
t.Fatalf("AVX512ER: expected %v, got %v", expected, got)
}
t.Log("AVX512ER Support:", got)
}
// TestAVX512CD tests AVX512CD() function (AVX-512 Conflict Detection Instructions)
func TestAVX512CD(t *testing.T) {
got := CPU.AVX512CD()
expected := CPU.Features&AVX512CD == AVX512CD
if got != expected {
t.Fatalf("AVX512CD: expected %v, got %v", expected, got)
}
t.Log("AVX512CD Support:", got)
}
// TestAVX512BW tests AVX512BW() function (AVX-512 Byte and Word Instructions)
func TestAVX512BW(t *testing.T) {
got := CPU.AVX512BW()
expected := CPU.Features&AVX512BW == AVX512BW
if got != expected {
t.Fatalf("AVX512BW: expected %v, got %v", expected, got)
}
t.Log("AVX512BW Support:", got)
}
// TestAVX512VL tests AVX512VL() function (AVX-512 Vector Length Extensions)
func TestAVX512VL(t *testing.T) {
got := CPU.AVX512VL()
expected := CPU.Features&AVX512VL == AVX512VL
if got != expected {
t.Fatalf("AVX512VL: expected %v, got %v", expected, got)
}
t.Log("AVX512VL Support:", got)
}
// TestAVX512VL tests AVX512VBMI() function (AVX-512 Vector Bit Manipulation Instructions)
func TestAVX512VBMI(t *testing.T) {
got := CPU.AVX512VBMI()
expected := CPU.Features&AVX512VBMI == AVX512VBMI
if got != expected {
t.Fatalf("AVX512VBMI: expected %v, got %v", expected, got)
}
t.Log("AVX512VBMI Support:", got)
}
// TestMPX tests MPX() function (Intel MPX (Memory Protection Extensions))
func TestMPX(t *testing.T) {
got := CPU.MPX()
expected := CPU.Features&MPX == MPX
if got != expected {
t.Fatalf("MPX: expected %v, got %v", expected, got)
}
t.Log("MPX Support:", got)
}
// TestERMS tests ERMS() function (Enhanced REP MOVSB/STOSB)
func TestERMS(t *testing.T) {
got := CPU.ERMS()
expected := CPU.Features&ERMS == ERMS
if got != expected {
t.Fatalf("ERMS: expected %v, got %v", expected, got)
}
t.Log("ERMS Support:", got)
}
// TestVendor writes the detected vendor. Will be 0 if unknown
func TestVendor(t *testing.T) {
t.Log("Vendor ID:", CPU.VendorID)
}
// Intel returns true if vendor is recognized as Intel
func TestIntel(t *testing.T) {
got := CPU.Intel()
expected := CPU.VendorID == Intel
if got != expected {
t.Fatalf("TestIntel: expected %v, got %v", expected, got)
}
t.Log("TestIntel:", got)
}
// AMD returns true if vendor is recognized as AMD
func TestAMD(t *testing.T) {
got := CPU.AMD()
expected := CPU.VendorID == AMD
if got != expected {
t.Fatalf("TestAMD: expected %v, got %v", expected, got)
}
t.Log("TestAMD:", got)
}
// Transmeta returns true if vendor is recognized as Transmeta
func TestTransmeta(t *testing.T) {
got := CPU.Transmeta()
expected := CPU.VendorID == Transmeta
if got != expected {
t.Fatalf("TestTransmeta: expected %v, got %v", expected, got)
}
t.Log("TestTransmeta:", got)
}
// NSC returns true if vendor is recognized as National Semiconductor
func TestNSC(t *testing.T) {
got := CPU.NSC()
expected := CPU.VendorID == NSC
if got != expected {
t.Fatalf("TestNSC: expected %v, got %v", expected, got)
}
t.Log("TestNSC:", got)
}
// VIA returns true if vendor is recognized as VIA
func TestVIA(t *testing.T) {
got := CPU.VIA()
expected := CPU.VendorID == VIA
if got != expected {
t.Fatalf("TestVIA: expected %v, got %v", expected, got)
}
t.Log("TestVIA:", got)
}
// Test VM function
func TestVM(t *testing.T) {
t.Log("Vendor ID:", CPU.VM())
}
// Test RTCounter function
func TestRtCounter(t *testing.T) {
a := CPU.RTCounter()
b := CPU.RTCounter()
t.Log("CPU Counter:", a, b, b-a)
}
// Prints the value of Ia32TscAux()
func TestIa32TscAux(t *testing.T) {
ecx := CPU.Ia32TscAux()
t.Logf("Ia32TscAux:0x%x\n", ecx)
if ecx != 0 {
chip := (ecx & 0xFFF000) >> 12
core := ecx & 0xFFF
t.Log("Likely chip, core:", chip, core)
}
}
func TestThreadsPerCoreNZ(t *testing.T) {
if CPU.ThreadsPerCore == 0 {
t.Fatal("threads per core is zero")
}
}
// Prints the value of LogicalCPU()
func TestLogicalCPU(t *testing.T) {
t.Log("Currently executing on cpu:", CPU.LogicalCPU())
}
func TestMaxFunction(t *testing.T) {
expect := maxFunctionID()
if CPU.maxFunc != expect {
t.Fatal("Max function does not match, expected", expect, "but got", CPU.maxFunc)
}
expect = maxExtendedFunction()
if CPU.maxExFunc != expect {
t.Fatal("Max Extended function does not match, expected", expect, "but got", CPU.maxFunc)
}
}
// This example will calculate the chip/core number on Linux
// Linux encodes numa id (<<12) and core id (8bit) into TSC_AUX.
func ExampleCPUInfo_Ia32TscAux(t *testing.T) {
ecx := CPU.Ia32TscAux()
if ecx == 0 {
fmt.Println("Unknown CPU ID")
return
}
chip := (ecx & 0xFFF000) >> 12
core := ecx & 0xFFF
fmt.Println("Chip, Core:", chip, core)
}
/*
func TestPhysical(t *testing.T) {
var test16 = "CPUID 00000000: 0000000d-756e6547-6c65746e-49656e69 \nCPUID 00000001: 000206d7-03200800-1fbee3ff-bfebfbff \nCPUID 00000002: 76035a01-00f0b2ff-00000000-00ca0000 \nCPUID 00000003: 00000000-00000000-00000000-00000000 \nCPUID 00000004: 3c004121-01c0003f-0000003f-00000000 \nCPUID 00000004: 3c004122-01c0003f-0000003f-00000000 \nCPUID 00000004: 3c004143-01c0003f-000001ff-00000000 \nCPUID 00000004: 3c07c163-04c0003f-00003fff-00000006 \nCPUID 00000005: 00000040-00000040-00000003-00021120 \nCPUID 00000006: 00000075-00000002-00000009-00000000 \nCPUID 00000007: 00000000-00000000-00000000-00000000 \nCPUID 00000008: 00000000-00000000-00000000-00000000 \nCPUID 00000009: 00000001-00000000-00000000-00000000 \nCPUID 0000000a: 07300403-00000000-00000000-00000603 \nCPUID 0000000b: 00000000-00000000-00000003-00000003 \nCPUID 0000000b: 00000005-00000010-00000201-00000003 \nCPUID 0000000c: 00000000-00000000-00000000-00000000 \nCPUID 0000000d: 00000007-00000340-00000340-00000000 \nCPUID 0000000d: 00000001-00000000-00000000-00000000 \nCPUID 0000000d: 00000100-00000240-00000000-00000000 \nCPUID 80000000: 80000008-00000000-00000000-00000000 \nCPUID 80000001: 00000000-00000000-00000001-2c100800 \nCPUID 80000002: 20202020-49202020-6c65746e-20295228 \nCPUID 80000003: 6e6f6558-20295228-20555043-322d3545 \nCPUID 80000004: 20303636-20402030-30322e32-007a4847 \nCPUID 80000005: 00000000-00000000-00000000-00000000 \nCPUID 80000006: 00000000-00000000-01006040-00000000 \nCPUID 80000007: 00000000-00000000-00000000-00000100 \nCPUID 80000008: 0000302e-00000000-00000000-00000000"
restore := mockCPU([]byte(test16))
Detect()
t.Log("Name:", CPU.BrandName)
n := maxFunctionID()
t.Logf("Max Function:0x%x\n", n)
n = maxExtendedFunction()
t.Logf("Max Extended Function:0x%x\n", n)
t.Log("PhysicalCores:", CPU.PhysicalCores)
t.Log("ThreadsPerCore:", CPU.ThreadsPerCore)
t.Log("LogicalCores:", CPU.LogicalCores)
t.Log("Family", CPU.Family, "Model:", CPU.Model)
t.Log("Features:", CPU.Features)
t.Log("Cacheline bytes:", CPU.CacheLine)
t.Log("L1 Instruction Cache:", CPU.Cache.L1I, "bytes")
t.Log("L1 Data Cache:", CPU.Cache.L1D, "bytes")
t.Log("L2 Cache:", CPU.Cache.L2, "bytes")
t.Log("L3 Cache:", CPU.Cache.L3, "bytes")
if CPU.LogicalCores > 0 && CPU.PhysicalCores > 0 {
if CPU.LogicalCores != CPU.PhysicalCores*CPU.ThreadsPerCore {
t.Fatalf("Core count mismatch, LogicalCores (%d) != PhysicalCores (%d) * CPU.ThreadsPerCore (%d)",
CPU.LogicalCores, CPU.PhysicalCores, CPU.ThreadsPerCore)
}
}
if CPU.ThreadsPerCore > 1 && !CPU.HTT() {
t.Fatalf("Hyperthreading not detected")
}
if CPU.ThreadsPerCore == 1 && CPU.HTT() {
t.Fatalf("Hyperthreading detected, but only 1 Thread per core")
}
restore()
Detect()
TestCPUID(t)
}
*/

17
vendor/github.com/klauspost/cpuid/detect_intel.go generated vendored
View File

@@ -1,17 +0,0 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo amd64,!gccgo
package cpuid
func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
func asmXgetbv(index uint32) (eax, edx uint32)
func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
func initCPU() {
cpuid = asmCpuid
cpuidex = asmCpuidex
xgetbv = asmXgetbv
rdtscpAsm = asmRdtscpAsm
}

23
vendor/github.com/klauspost/cpuid/detect_ref.go generated vendored
View File

@@ -1,23 +0,0 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build !amd64,!386 gccgo
package cpuid
func initCPU() {
cpuid = func(op uint32) (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
cpuidex = func(op, op2 uint32) (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
xgetbv = func(index uint32) (eax, edx uint32) {
return 0, 0
}
rdtscpAsm = func() (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
}

3
vendor/github.com/klauspost/cpuid/generate.go generated vendored
View File

@@ -1,3 +0,0 @@
package cpuid
//go:generate go run private-gen.go

209
vendor/github.com/klauspost/cpuid/mockcpu_test.go generated vendored
View File

@@ -1,209 +0,0 @@
package cpuid
import (
"archive/zip"
"fmt"
"io/ioutil"
"sort"
"strings"
"testing"
)
type fakecpuid map[uint32][][]uint32
type idfuncs struct {
cpuid func(op uint32) (eax, ebx, ecx, edx uint32)
cpuidex func(op, op2 uint32) (eax, ebx, ecx, edx uint32)
xgetbv func(index uint32) (eax, edx uint32)
}
func (f fakecpuid) String() string {
var out = make([]string, 0, len(f))
for key, val := range f {
for _, v := range val {
out = append(out, fmt.Sprintf("CPUID %08x: [%08x, %08x, %08x, %08x]", key, v[0], v[1], v[2], v[3]))
}
}
sorter := sort.StringSlice(out)
sort.Sort(&sorter)
return strings.Join(sorter, "\n")
}
func mockCPU(def []byte) func() {
lines := strings.Split(string(def), "\n")
anyfound := false
fakeID := make(fakecpuid)
for _, line := range lines {
line = strings.Trim(line, "\r\t ")
if !strings.HasPrefix(line, "CPUID") {
continue
}
// Only collect for first cpu
if strings.HasPrefix(line, "CPUID 00000000") {
if anyfound {
break
}
}
if !strings.Contains(line, "-") {
//continue
}
items := strings.Split(line, ":")
if len(items) < 2 {
if len(line) == 51 || len(line) == 50 {
items = []string{line[0:14], line[15:]}
} else {
items = strings.Split(line, "\t")
if len(items) != 2 {
//fmt.Println("not found:", line, "len:", len(line))
continue
}
}
}
items = items[0:2]
vals := strings.Trim(items[1], "\r\n ")
var idV uint32
n, err := fmt.Sscanf(items[0], "CPUID %x", &idV)
if err != nil || n != 1 {
continue
}
existing, ok := fakeID[idV]
if !ok {
existing = make([][]uint32, 0)
}
values := make([]uint32, 4)
n, err = fmt.Sscanf(vals, "%x-%x-%x-%x", &values[0], &values[1], &values[2], &values[3])
if n != 4 || err != nil {
n, err = fmt.Sscanf(vals, "%x %x %x %x", &values[0], &values[1], &values[2], &values[3])
if n != 4 || err != nil {
//fmt.Println("scanned", vals, "got", n, "Err:", err)
continue
}
}
existing = append(existing, values)
fakeID[idV] = existing
anyfound = true
}
restorer := func(f idfuncs) func() {
return func() {
cpuid = f.cpuid
cpuidex = f.cpuidex
xgetbv = f.xgetbv
}
}(idfuncs{cpuid: cpuid, cpuidex: cpuidex, xgetbv: xgetbv})
cpuid = func(op uint32) (eax, ebx, ecx, edx uint32) {
if op == 0x80000000 || op == 0 {
var ok bool
_, ok = fakeID[op]
if !ok {
return 0, 0, 0, 0
}
}
first, ok := fakeID[op]
if !ok {
if op > maxFunctionID() {
panic(fmt.Sprintf("Base not found: %v, request:%#v\n", fakeID, op))
} else {
// we have some entries missing
return 0, 0, 0, 0
}
}
theid := first[0]
return theid[0], theid[1], theid[2], theid[3]
}
cpuidex = func(op, op2 uint32) (eax, ebx, ecx, edx uint32) {
if op == 0x80000000 {
var ok bool
_, ok = fakeID[op]
if !ok {
return 0, 0, 0, 0
}
}
first, ok := fakeID[op]
if !ok {
if op > maxExtendedFunction() {
panic(fmt.Sprintf("Extended not found Info: %v, request:%#v, %#v\n", fakeID, op, op2))
} else {
// we have some entries missing
return 0, 0, 0, 0
}
}
if int(op2) >= len(first) {
//fmt.Printf("Extended not found Info: %v, request:%#v, %#v\n", fakeID, op, op2)
return 0, 0, 0, 0
}
theid := first[op2]
return theid[0], theid[1], theid[2], theid[3]
}
xgetbv = func(index uint32) (eax, edx uint32) {
first, ok := fakeID[1]
if !ok {
panic(fmt.Sprintf("XGETBV not supported %v", fakeID))
}
second := first[0]
// ECX bit 26 must be set
if (second[2] & 1 << 26) == 0 {
panic(fmt.Sprintf("XGETBV not supported %v", fakeID))
}
// We don't have any data to return, unfortunately
return 0, 0
}
return restorer
}
func TestMocks(t *testing.T) {
zr, err := zip.OpenReader("testdata/cpuid_data.zip")
if err != nil {
t.Skip("No testdata:", err)
}
defer zr.Close()
for _, f := range zr.File {
rc, err := f.Open()
if err != nil {
t.Fatal(err)
}
content, err := ioutil.ReadAll(rc)
if err != nil {
t.Fatal(err)
}
rc.Close()
t.Log("Opening", f.FileInfo().Name())
restore := mockCPU(content)
Detect()
t.Log("Name:", CPU.BrandName)
n := maxFunctionID()
t.Logf("Max Function:0x%x\n", n)
n = maxExtendedFunction()
t.Logf("Max Extended Function:0x%x\n", n)
t.Log("PhysicalCores:", CPU.PhysicalCores)
t.Log("ThreadsPerCore:", CPU.ThreadsPerCore)
t.Log("LogicalCores:", CPU.LogicalCores)
t.Log("Family", CPU.Family, "Model:", CPU.Model)
t.Log("Features:", CPU.Features)
t.Log("Cacheline bytes:", CPU.CacheLine)
t.Log("L1 Instruction Cache:", CPU.Cache.L1I, "bytes")
t.Log("L1 Data Cache:", CPU.Cache.L1D, "bytes")
t.Log("L2 Cache:", CPU.Cache.L2, "bytes")
t.Log("L3 Cache:", CPU.Cache.L3, "bytes")
if CPU.LogicalCores > 0 && CPU.PhysicalCores > 0 {
if CPU.LogicalCores != CPU.PhysicalCores*CPU.ThreadsPerCore {
t.Fatalf("Core count mismatch, LogicalCores (%d) != PhysicalCores (%d) * CPU.ThreadsPerCore (%d)",
CPU.LogicalCores, CPU.PhysicalCores, CPU.ThreadsPerCore)
}
}
if CPU.ThreadsPerCore > 1 && !CPU.HTT() {
t.Fatalf("Hyperthreading not detected")
}
if CPU.ThreadsPerCore == 1 && CPU.HTT() {
t.Fatalf("Hyperthreading detected, but only 1 Thread per core")
}
restore()
}
Detect()
}

476
vendor/github.com/klauspost/cpuid/private-gen.go generated vendored
View File

@@ -1,476 +0,0 @@
// +build ignore
package main
import (
"bytes"
"fmt"
"go/ast"
"go/parser"
"go/printer"
"go/token"
"io"
"io/ioutil"
"log"
"os"
"reflect"
"strings"
"unicode"
"unicode/utf8"
)
var inFiles = []string{"cpuid.go", "cpuid_test.go"}
var copyFiles = []string{"cpuid_amd64.s", "cpuid_386.s", "detect_ref.go", "detect_intel.go"}
var fileSet = token.NewFileSet()
var reWrites = []rewrite{
initRewrite("CPUInfo -> cpuInfo"),
initRewrite("Vendor -> vendor"),
initRewrite("Flags -> flags"),
initRewrite("Detect -> detect"),
initRewrite("CPU -> cpu"),
}
var excludeNames = map[string]bool{"string": true, "join": true, "trim": true,
// cpuid_test.go
"t": true, "println": true, "logf": true, "log": true, "fatalf": true, "fatal": true,
}
var excludePrefixes = []string{"test", "benchmark"}
func main() {
Package := "private"
parserMode := parser.ParseComments
exported := make(map[string]rewrite)
for _, file := range inFiles {
in, err := os.Open(file)
if err != nil {
log.Fatalf("opening input", err)
}
src, err := ioutil.ReadAll(in)
if err != nil {
log.Fatalf("reading input", err)
}
astfile, err := parser.ParseFile(fileSet, file, src, parserMode)
if err != nil {
log.Fatalf("parsing input", err)
}
for _, rw := range reWrites {
astfile = rw(astfile)
}
// Inspect the AST and print all identifiers and literals.
var startDecl token.Pos
var endDecl token.Pos
ast.Inspect(astfile, func(n ast.Node) bool {
var s string
switch x := n.(type) {
case *ast.Ident:
if x.IsExported() {
t := strings.ToLower(x.Name)
for _, pre := range excludePrefixes {
if strings.HasPrefix(t, pre) {
return true
}
}
if excludeNames[t] != true {
//if x.Pos() > startDecl && x.Pos() < endDecl {
exported[x.Name] = initRewrite(x.Name + " -> " + t)
}
}
case *ast.GenDecl:
if x.Tok == token.CONST && x.Lparen > 0 {
startDecl = x.Lparen
endDecl = x.Rparen
// fmt.Printf("Decl:%s -> %s\n", fileSet.Position(startDecl), fileSet.Position(endDecl))
}
}
if s != "" {
fmt.Printf("%s:\t%s\n", fileSet.Position(n.Pos()), s)
}
return true
})
for _, rw := range exported {
astfile = rw(astfile)
}
var buf bytes.Buffer
printer.Fprint(&buf, fileSet, astfile)
// Remove package documentation and insert information
s := buf.String()
ind := strings.Index(buf.String(), "\npackage cpuid")
s = s[ind:]
s = "// Generated, DO NOT EDIT,\n" +
"// but copy it to your own project and rename the package.\n" +
"// See more at http://github.com/klauspost/cpuid\n" +
s
outputName := Package + string(os.PathSeparator) + file
err = ioutil.WriteFile(outputName, []byte(s), 0644)
if err != nil {
log.Fatalf("writing output: %s", err)
}
log.Println("Generated", outputName)
}
for _, file := range copyFiles {
dst := ""
if strings.HasPrefix(file, "cpuid") {
dst = Package + string(os.PathSeparator) + file
} else {
dst = Package + string(os.PathSeparator) + "cpuid_" + file
}
err := copyFile(file, dst)
if err != nil {
log.Fatalf("copying file: %s", err)
}
log.Println("Copied", dst)
}
}
// CopyFile copies a file from src to dst. If src and dst files exist, and are
// the same, then return success. Copy the file contents from src to dst.
func copyFile(src, dst string) (err error) {
sfi, err := os.Stat(src)
if err != nil {
return
}
if !sfi.Mode().IsRegular() {
// cannot copy non-regular files (e.g., directories,
// symlinks, devices, etc.)
return fmt.Errorf("CopyFile: non-regular source file %s (%q)", sfi.Name(), sfi.Mode().String())
}
dfi, err := os.Stat(dst)
if err != nil {
if !os.IsNotExist(err) {
return
}
} else {
if !(dfi.Mode().IsRegular()) {
return fmt.Errorf("CopyFile: non-regular destination file %s (%q)", dfi.Name(), dfi.Mode().String())
}
if os.SameFile(sfi, dfi) {
return
}
}
err = copyFileContents(src, dst)
return
}
// copyFileContents copies the contents of the file named src to the file named
// by dst. The file will be created if it does not already exist. If the
// destination file exists, all it's contents will be replaced by the contents
// of the source file.
func copyFileContents(src, dst string) (err error) {
in, err := os.Open(src)
if err != nil {
return
}
defer in.Close()
out, err := os.Create(dst)
if err != nil {
return
}
defer func() {
cerr := out.Close()
if err == nil {
err = cerr
}
}()
if _, err = io.Copy(out, in); err != nil {
return
}
err = out.Sync()
return
}
type rewrite func(*ast.File) *ast.File
// Mostly copied from gofmt
func initRewrite(rewriteRule string) rewrite {
f := strings.Split(rewriteRule, "->")
if len(f) != 2 {
fmt.Fprintf(os.Stderr, "rewrite rule must be of the form 'pattern -> replacement'\n")
os.Exit(2)
}
pattern := parseExpr(f[0], "pattern")
replace := parseExpr(f[1], "replacement")
return func(p *ast.File) *ast.File { return rewriteFile(pattern, replace, p) }
}
// parseExpr parses s as an expression.
// It might make sense to expand this to allow statement patterns,
// but there are problems with preserving formatting and also
// with what a wildcard for a statement looks like.
func parseExpr(s, what string) ast.Expr {
x, err := parser.ParseExpr(s)
if err != nil {
fmt.Fprintf(os.Stderr, "parsing %s %s at %s\n", what, s, err)
os.Exit(2)
}
return x
}
// Keep this function for debugging.
/*
func dump(msg string, val reflect.Value) {
fmt.Printf("%s:\n", msg)
ast.Print(fileSet, val.Interface())
fmt.Println()
}
*/
// rewriteFile applies the rewrite rule 'pattern -> replace' to an entire file.
func rewriteFile(pattern, replace ast.Expr, p *ast.File) *ast.File {
cmap := ast.NewCommentMap(fileSet, p, p.Comments)
m := make(map[string]reflect.Value)
pat := reflect.ValueOf(pattern)
repl := reflect.ValueOf(replace)
var rewriteVal func(val reflect.Value) reflect.Value
rewriteVal = func(val reflect.Value) reflect.Value {
// don't bother if val is invalid to start with
if !val.IsValid() {
return reflect.Value{}
}
for k := range m {
delete(m, k)
}
val = apply(rewriteVal, val)
if match(m, pat, val) {
val = subst(m, repl, reflect.ValueOf(val.Interface().(ast.Node).Pos()))
}
return val
}
r := apply(rewriteVal, reflect.ValueOf(p)).Interface().(*ast.File)
r.Comments = cmap.Filter(r).Comments() // recreate comments list
return r
}
// set is a wrapper for x.Set(y); it protects the caller from panics if x cannot be changed to y.
func set(x, y reflect.Value) {
// don't bother if x cannot be set or y is invalid
if !x.CanSet() || !y.IsValid() {
return
}
defer func() {
if x := recover(); x != nil {
if s, ok := x.(string); ok &&
(strings.Contains(s, "type mismatch") || strings.Contains(s, "not assignable")) {
// x cannot be set to y - ignore this rewrite
return
}
panic(x)
}
}()
x.Set(y)
}
// Values/types for special cases.
var (
objectPtrNil = reflect.ValueOf((*ast.Object)(nil))
scopePtrNil = reflect.ValueOf((*ast.Scope)(nil))
identType = reflect.TypeOf((*ast.Ident)(nil))
objectPtrType = reflect.TypeOf((*ast.Object)(nil))
positionType = reflect.TypeOf(token.NoPos)
callExprType = reflect.TypeOf((*ast.CallExpr)(nil))
scopePtrType = reflect.TypeOf((*ast.Scope)(nil))
)
// apply replaces each AST field x in val with f(x), returning val.
// To avoid extra conversions, f operates on the reflect.Value form.
func apply(f func(reflect.Value) reflect.Value, val reflect.Value) reflect.Value {
if !val.IsValid() {
return reflect.Value{}
}
// *ast.Objects introduce cycles and are likely incorrect after
// rewrite; don't follow them but replace with nil instead
if val.Type() == objectPtrType {
return objectPtrNil
}
// similarly for scopes: they are likely incorrect after a rewrite;
// replace them with nil
if val.Type() == scopePtrType {
return scopePtrNil
}
switch v := reflect.Indirect(val); v.Kind() {
case reflect.Slice:
for i := 0; i < v.Len(); i++ {
e := v.Index(i)
set(e, f(e))
}
case reflect.Struct:
for i := 0; i < v.NumField(); i++ {
e := v.Field(i)
set(e, f(e))
}
case reflect.Interface:
e := v.Elem()
set(v, f(e))
}
return val
}
func isWildcard(s string) bool {
rune, size := utf8.DecodeRuneInString(s)
return size == len(s) && unicode.IsLower(rune)
}
// match returns true if pattern matches val,
// recording wildcard submatches in m.
// If m == nil, match checks whether pattern == val.
func match(m map[string]reflect.Value, pattern, val reflect.Value) bool {
// Wildcard matches any expression. If it appears multiple
// times in the pattern, it must match the same expression
// each time.
if m != nil && pattern.IsValid() && pattern.Type() == identType {
name := pattern.Interface().(*ast.Ident).Name
if isWildcard(name) && val.IsValid() {
// wildcards only match valid (non-nil) expressions.
if _, ok := val.Interface().(ast.Expr); ok && !val.IsNil() {
if old, ok := m[name]; ok {
return match(nil, old, val)
}
m[name] = val
return true
}
}
}
// Otherwise, pattern and val must match recursively.
if !pattern.IsValid() || !val.IsValid() {
return !pattern.IsValid() && !val.IsValid()
}
if pattern.Type() != val.Type() {
return false
}
// Special cases.
switch pattern.Type() {
case identType:
// For identifiers, only the names need to match
// (and none of the other *ast.Object information).
// This is a common case, handle it all here instead
// of recursing down any further via reflection.
p := pattern.Interface().(*ast.Ident)
v := val.Interface().(*ast.Ident)
return p == nil && v == nil || p != nil && v != nil && p.Name == v.Name
case objectPtrType, positionType:
// object pointers and token positions always match
return true
case callExprType:
// For calls, the Ellipsis fields (token.Position) must
// match since that is how f(x) and f(x...) are different.
// Check them here but fall through for the remaining fields.
p := pattern.Interface().(*ast.CallExpr)
v := val.Interface().(*ast.CallExpr)
if p.Ellipsis.IsValid() != v.Ellipsis.IsValid() {
return false
}
}
p := reflect.Indirect(pattern)
v := reflect.Indirect(val)
if !p.IsValid() || !v.IsValid() {
return !p.IsValid() && !v.IsValid()
}
switch p.Kind() {
case reflect.Slice:
if p.Len() != v.Len() {
return false
}
for i := 0; i < p.Len(); i++ {
if !match(m, p.Index(i), v.Index(i)) {
return false
}
}
return true
case reflect.Struct:
for i := 0; i < p.NumField(); i++ {
if !match(m, p.Field(i), v.Field(i)) {
return false
}
}
return true
case reflect.Interface:
return match(m, p.Elem(), v.Elem())
}
// Handle token integers, etc.
return p.Interface() == v.Interface()
}
// subst returns a copy of pattern with values from m substituted in place
// of wildcards and pos used as the position of tokens from the pattern.
// if m == nil, subst returns a copy of pattern and doesn't change the line
// number information.
func subst(m map[string]reflect.Value, pattern reflect.Value, pos reflect.Value) reflect.Value {
if !pattern.IsValid() {
return reflect.Value{}
}
// Wildcard gets replaced with map value.
if m != nil && pattern.Type() == identType {
name := pattern.Interface().(*ast.Ident).Name
if isWildcard(name) {
if old, ok := m[name]; ok {
return subst(nil, old, reflect.Value{})
}
}
}
if pos.IsValid() && pattern.Type() == positionType {
// use new position only if old position was valid in the first place
if old := pattern.Interface().(token.Pos); !old.IsValid() {
return pattern
}
return pos
}
// Otherwise copy.
switch p := pattern; p.Kind() {
case reflect.Slice:
v := reflect.MakeSlice(p.Type(), p.Len(), p.Len())
for i := 0; i < p.Len(); i++ {
v.Index(i).Set(subst(m, p.Index(i), pos))
}
return v
case reflect.Struct:
v := reflect.New(p.Type()).Elem()
for i := 0; i < p.NumField(); i++ {
v.Field(i).Set(subst(m, p.Field(i), pos))
}
return v
case reflect.Ptr:
v := reflect.New(p.Type()).Elem()
if elem := p.Elem(); elem.IsValid() {
v.Set(subst(m, elem, pos).Addr())
}
return v
case reflect.Interface:
v := reflect.New(p.Type()).Elem()
if elem := p.Elem(); elem.IsValid() {
v.Set(subst(m, elem, pos))
}
return v
}
return pattern
}

6
vendor/github.com/klauspost/cpuid/private/README.md generated vendored
View File

@@ -1,6 +0,0 @@
# cpuid private
This is a specially converted of the cpuid package, so it can be included in
a package without exporting anything.
Package home: https://github.com/klauspost/cpuid

987
vendor/github.com/klauspost/cpuid/private/cpuid.go generated vendored
View File

@@ -1,987 +0,0 @@
// Generated, DO NOT EDIT,
// but copy it to your own project and rename the package.
// See more at http://github.com/klauspost/cpuid
package cpuid
import (
"strings"
)
// Vendor is a representation of a CPU vendor.
type vendor int
const (
other vendor = iota
intel
amd
via
transmeta
nsc
kvm // Kernel-based Virtual Machine
msvm // Microsoft Hyper-V or Windows Virtual PC
vmware
xenhvm
)
const (
cmov = 1 << iota // i686 CMOV
nx // NX (No-Execute) bit
amd3dnow // AMD 3DNOW
amd3dnowext // AMD 3DNowExt
mmx // standard MMX
mmxext // SSE integer functions or AMD MMX ext
sse // SSE functions
sse2 // P4 SSE functions
sse3 // Prescott SSE3 functions
ssse3 // Conroe SSSE3 functions
sse4 // Penryn SSE4.1 functions
sse4a // AMD Barcelona microarchitecture SSE4a instructions
sse42 // Nehalem SSE4.2 functions
avx // AVX functions
avx2 // AVX2 functions
fma3 // Intel FMA 3
fma4 // Bulldozer FMA4 functions
xop // Bulldozer XOP functions
f16c // Half-precision floating-point conversion
bmi1 // Bit Manipulation Instruction Set 1
bmi2 // Bit Manipulation Instruction Set 2
tbm // AMD Trailing Bit Manipulation
lzcnt // LZCNT instruction
popcnt // POPCNT instruction
aesni // Advanced Encryption Standard New Instructions
clmul // Carry-less Multiplication
htt // Hyperthreading (enabled)
hle // Hardware Lock Elision
rtm // Restricted Transactional Memory
rdrand // RDRAND instruction is available
rdseed // RDSEED instruction is available
adx // Intel ADX (Multi-Precision Add-Carry Instruction Extensions)
sha // Intel SHA Extensions
avx512f // AVX-512 Foundation
avx512dq // AVX-512 Doubleword and Quadword Instructions
avx512ifma // AVX-512 Integer Fused Multiply-Add Instructions
avx512pf // AVX-512 Prefetch Instructions
avx512er // AVX-512 Exponential and Reciprocal Instructions
avx512cd // AVX-512 Conflict Detection Instructions
avx512bw // AVX-512 Byte and Word Instructions
avx512vl // AVX-512 Vector Length Extensions
avx512vbmi // AVX-512 Vector Bit Manipulation Instructions
mpx // Intel MPX (Memory Protection Extensions)
erms // Enhanced REP MOVSB/STOSB
rdtscp // RDTSCP Instruction
cx16 // CMPXCHG16B Instruction
// Performance indicators
sse2slow // SSE2 is supported, but usually not faster
sse3slow // SSE3 is supported, but usually not faster
atom // Atom processor, some SSSE3 instructions are slower
)
var flagNames = map[flags]string{
cmov: "CMOV", // i686 CMOV
nx: "NX", // NX (No-Execute) bit
amd3dnow: "AMD3DNOW", // AMD 3DNOW
amd3dnowext: "AMD3DNOWEXT", // AMD 3DNowExt
mmx: "MMX", // Standard MMX
mmxext: "MMXEXT", // SSE integer functions or AMD MMX ext
sse: "SSE", // SSE functions
sse2: "SSE2", // P4 SSE2 functions
sse3: "SSE3", // Prescott SSE3 functions
ssse3: "SSSE3", // Conroe SSSE3 functions
sse4: "SSE4.1", // Penryn SSE4.1 functions
sse4a: "SSE4A", // AMD Barcelona microarchitecture SSE4a instructions
sse42: "SSE4.2", // Nehalem SSE4.2 functions
avx: "AVX", // AVX functions
avx2: "AVX2", // AVX functions
fma3: "FMA3", // Intel FMA 3
fma4: "FMA4", // Bulldozer FMA4 functions
xop: "XOP", // Bulldozer XOP functions
f16c: "F16C", // Half-precision floating-point conversion
bmi1: "BMI1", // Bit Manipulation Instruction Set 1
bmi2: "BMI2", // Bit Manipulation Instruction Set 2
tbm: "TBM", // AMD Trailing Bit Manipulation
lzcnt: "LZCNT", // LZCNT instruction
popcnt: "POPCNT", // POPCNT instruction
aesni: "AESNI", // Advanced Encryption Standard New Instructions
clmul: "CLMUL", // Carry-less Multiplication
htt: "HTT", // Hyperthreading (enabled)
hle: "HLE", // Hardware Lock Elision
rtm: "RTM", // Restricted Transactional Memory
rdrand: "RDRAND", // RDRAND instruction is available
rdseed: "RDSEED", // RDSEED instruction is available
adx: "ADX", // Intel ADX (Multi-Precision Add-Carry Instruction Extensions)
sha: "SHA", // Intel SHA Extensions
avx512f: "AVX512F", // AVX-512 Foundation
avx512dq: "AVX512DQ", // AVX-512 Doubleword and Quadword Instructions
avx512ifma: "AVX512IFMA", // AVX-512 Integer Fused Multiply-Add Instructions
avx512pf: "AVX512PF", // AVX-512 Prefetch Instructions
avx512er: "AVX512ER", // AVX-512 Exponential and Reciprocal Instructions
avx512cd: "AVX512CD", // AVX-512 Conflict Detection Instructions
avx512bw: "AVX512BW", // AVX-512 Byte and Word Instructions
avx512vl: "AVX512VL", // AVX-512 Vector Length Extensions
avx512vbmi: "AVX512VBMI", // AVX-512 Vector Bit Manipulation Instructions
mpx: "MPX", // Intel MPX (Memory Protection Extensions)
erms: "ERMS", // Enhanced REP MOVSB/STOSB
rdtscp: "RDTSCP", // RDTSCP Instruction
cx16: "CX16", // CMPXCHG16B Instruction
// Performance indicators
sse2slow: "SSE2SLOW", // SSE2 supported, but usually not faster
sse3slow: "SSE3SLOW", // SSE3 supported, but usually not faster
atom: "ATOM", // Atom processor, some SSSE3 instructions are slower
}
// CPUInfo contains information about the detected system CPU.
type cpuInfo struct {
brandname string // Brand name reported by the CPU
vendorid vendor // Comparable CPU vendor ID
features flags // Features of the CPU
physicalcores int // Number of physical processor cores in your CPU. Will be 0 if undetectable.
threadspercore int // Number of threads per physical core. Will be 1 if undetectable.
logicalcores int // Number of physical cores times threads that can run on each core through the use of hyperthreading. Will be 0 if undetectable.
family int // CPU family number
model int // CPU model number
cacheline int // Cache line size in bytes. Will be 0 if undetectable.
cache struct {
l1i int // L1 Instruction Cache (per core or shared). Will be -1 if undetected
l1d int // L1 Data Cache (per core or shared). Will be -1 if undetected
l2 int // L2 Cache (per core or shared). Will be -1 if undetected
l3 int // L3 Instruction Cache (per core or shared). Will be -1 if undetected
}
maxFunc uint32
maxExFunc uint32
}
var cpuid func(op uint32) (eax, ebx, ecx, edx uint32)
var cpuidex func(op, op2 uint32) (eax, ebx, ecx, edx uint32)
var xgetbv func(index uint32) (eax, edx uint32)
var rdtscpAsm func() (eax, ebx, ecx, edx uint32)
// CPU contains information about the CPU as detected on startup,
// or when Detect last was called.
//
// Use this as the primary entry point to you data,
// this way queries are
var cpu cpuInfo
func init() {
initCPU()
detect()
}
// Detect will re-detect current CPU info.
// This will replace the content of the exported CPU variable.
//
// Unless you expect the CPU to change while you are running your program
// you should not need to call this function.
// If you call this, you must ensure that no other goroutine is accessing the
// exported CPU variable.
func detect() {
cpu.maxFunc = maxFunctionID()
cpu.maxExFunc = maxExtendedFunction()
cpu.brandname = brandName()
cpu.cacheline = cacheLine()
cpu.family, cpu.model = familyModel()
cpu.features = support()
cpu.threadspercore = threadsPerCore()
cpu.logicalcores = logicalCores()
cpu.physicalcores = physicalCores()
cpu.vendorid = vendorID()
cpu.cacheSize()
}
// Generated here: http://play.golang.org/p/BxFH2Gdc0G
// Cmov indicates support of CMOV instructions
func (c cpuInfo) cmov() bool {
return c.features&cmov != 0
}
// Amd3dnow indicates support of AMD 3DNOW! instructions
func (c cpuInfo) amd3dnow() bool {
return c.features&amd3dnow != 0
}
// Amd3dnowExt indicates support of AMD 3DNOW! Extended instructions
func (c cpuInfo) amd3dnowext() bool {
return c.features&amd3dnowext != 0
}
// MMX indicates support of MMX instructions
func (c cpuInfo) mmx() bool {
return c.features&mmx != 0
}
// MMXExt indicates support of MMXEXT instructions
// (SSE integer functions or AMD MMX ext)
func (c cpuInfo) mmxext() bool {
return c.features&mmxext != 0
}
// SSE indicates support of SSE instructions
func (c cpuInfo) sse() bool {
return c.features&sse != 0
}
// SSE2 indicates support of SSE 2 instructions
func (c cpuInfo) sse2() bool {
return c.features&sse2 != 0
}
// SSE3 indicates support of SSE 3 instructions
func (c cpuInfo) sse3() bool {
return c.features&sse3 != 0
}
// SSSE3 indicates support of SSSE 3 instructions
func (c cpuInfo) ssse3() bool {
return c.features&ssse3 != 0
}
// SSE4 indicates support of SSE 4 (also called SSE 4.1) instructions
func (c cpuInfo) sse4() bool {
return c.features&sse4 != 0
}
// SSE42 indicates support of SSE4.2 instructions
func (c cpuInfo) sse42() bool {
return c.features&sse42 != 0
}
// AVX indicates support of AVX instructions
// and operating system support of AVX instructions
func (c cpuInfo) avx() bool {
return c.features&avx != 0
}
// AVX2 indicates support of AVX2 instructions
func (c cpuInfo) avx2() bool {
return c.features&avx2 != 0
}
// FMA3 indicates support of FMA3 instructions
func (c cpuInfo) fma3() bool {
return c.features&fma3 != 0
}
// FMA4 indicates support of FMA4 instructions
func (c cpuInfo) fma4() bool {
return c.features&fma4 != 0
}
// XOP indicates support of XOP instructions
func (c cpuInfo) xop() bool {
return c.features&xop != 0
}
// F16C indicates support of F16C instructions
func (c cpuInfo) f16c() bool {
return c.features&f16c != 0
}
// BMI1 indicates support of BMI1 instructions
func (c cpuInfo) bmi1() bool {
return c.features&bmi1 != 0
}
// BMI2 indicates support of BMI2 instructions
func (c cpuInfo) bmi2() bool {
return c.features&bmi2 != 0
}
// TBM indicates support of TBM instructions
// (AMD Trailing Bit Manipulation)
func (c cpuInfo) tbm() bool {
return c.features&tbm != 0
}
// Lzcnt indicates support of LZCNT instruction
func (c cpuInfo) lzcnt() bool {
return c.features&lzcnt != 0
}
// Popcnt indicates support of POPCNT instruction
func (c cpuInfo) popcnt() bool {
return c.features&popcnt != 0
}
// HTT indicates the processor has Hyperthreading enabled
func (c cpuInfo) htt() bool {
return c.features&htt != 0
}
// SSE2Slow indicates that SSE2 may be slow on this processor
func (c cpuInfo) sse2slow() bool {
return c.features&sse2slow != 0
}
// SSE3Slow indicates that SSE3 may be slow on this processor
func (c cpuInfo) sse3slow() bool {
return c.features&sse3slow != 0
}
// AesNi indicates support of AES-NI instructions
// (Advanced Encryption Standard New Instructions)
func (c cpuInfo) aesni() bool {
return c.features&aesni != 0
}
// Clmul indicates support of CLMUL instructions
// (Carry-less Multiplication)
func (c cpuInfo) clmul() bool {
return c.features&clmul != 0
}
// NX indicates support of NX (No-Execute) bit
func (c cpuInfo) nx() bool {
return c.features&nx != 0
}
// SSE4A indicates support of AMD Barcelona microarchitecture SSE4a instructions
func (c cpuInfo) sse4a() bool {
return c.features&sse4a != 0
}
// HLE indicates support of Hardware Lock Elision
func (c cpuInfo) hle() bool {
return c.features&hle != 0
}
// RTM indicates support of Restricted Transactional Memory
func (c cpuInfo) rtm() bool {
return c.features&rtm != 0
}
// Rdrand indicates support of RDRAND instruction is available
func (c cpuInfo) rdrand() bool {
return c.features&rdrand != 0
}
// Rdseed indicates support of RDSEED instruction is available
func (c cpuInfo) rdseed() bool {
return c.features&rdseed != 0
}
// ADX indicates support of Intel ADX (Multi-Precision Add-Carry Instruction Extensions)
func (c cpuInfo) adx() bool {
return c.features&adx != 0
}
// SHA indicates support of Intel SHA Extensions
func (c cpuInfo) sha() bool {
return c.features&sha != 0
}
// AVX512F indicates support of AVX-512 Foundation
func (c cpuInfo) avx512f() bool {
return c.features&avx512f != 0
}
// AVX512DQ indicates support of AVX-512 Doubleword and Quadword Instructions
func (c cpuInfo) avx512dq() bool {
return c.features&avx512dq != 0
}
// AVX512IFMA indicates support of AVX-512 Integer Fused Multiply-Add Instructions
func (c cpuInfo) avx512ifma() bool {
return c.features&avx512ifma != 0
}
// AVX512PF indicates support of AVX-512 Prefetch Instructions
func (c cpuInfo) avx512pf() bool {
return c.features&avx512pf != 0
}
// AVX512ER indicates support of AVX-512 Exponential and Reciprocal Instructions
func (c cpuInfo) avx512er() bool {
return c.features&avx512er != 0
}
// AVX512CD indicates support of AVX-512 Conflict Detection Instructions
func (c cpuInfo) avx512cd() bool {
return c.features&avx512cd != 0
}
// AVX512BW indicates support of AVX-512 Byte and Word Instructions
func (c cpuInfo) avx512bw() bool {
return c.features&avx512bw != 0
}
// AVX512VL indicates support of AVX-512 Vector Length Extensions
func (c cpuInfo) avx512vl() bool {
return c.features&avx512vl != 0
}
// AVX512VBMI indicates support of AVX-512 Vector Bit Manipulation Instructions
func (c cpuInfo) avx512vbmi() bool {
return c.features&avx512vbmi != 0
}
// MPX indicates support of Intel MPX (Memory Protection Extensions)
func (c cpuInfo) mpx() bool {
return c.features&mpx != 0
}
// ERMS indicates support of Enhanced REP MOVSB/STOSB
func (c cpuInfo) erms() bool {
return c.features&erms != 0
}
func (c cpuInfo) rdtscp() bool {
return c.features&rdtscp != 0
}
func (c cpuInfo) cx16() bool {
return c.features&cx16 != 0
}
// Atom indicates an Atom processor
func (c cpuInfo) atom() bool {
return c.features&atom != 0
}
// Intel returns true if vendor is recognized as Intel
func (c cpuInfo) intel() bool {
return c.vendorid == intel
}
// AMD returns true if vendor is recognized as AMD
func (c cpuInfo) amd() bool {
return c.vendorid == amd
}
// Transmeta returns true if vendor is recognized as Transmeta
func (c cpuInfo) transmeta() bool {
return c.vendorid == transmeta
}
// NSC returns true if vendor is recognized as National Semiconductor
func (c cpuInfo) nsc() bool {
return c.vendorid == nsc
}
// VIA returns true if vendor is recognized as VIA
func (c cpuInfo) via() bool {
return c.vendorid == via
}
// RTCounter returns the 64-bit time-stamp counter
// Uses the RDTSCP instruction. The value 0 is returned
// if the CPU does not support the instruction.
func (c cpuInfo) rtcounter() uint64 {
if !c.rdtscp() {
return 0
}
a, _, _, d := rdtscpAsm()
return uint64(a) | (uint64(d) << 32)
}
// Ia32TscAux returns the IA32_TSC_AUX part of the RDTSCP.
// This variable is OS dependent, but on Linux contains information
// about the current cpu/core the code is running on.
// If the RDTSCP instruction isn't supported on the CPU, the value 0 is returned.
func (c cpuInfo) ia32tscaux() uint32 {
if !c.rdtscp() {
return 0
}
_, _, ecx, _ := rdtscpAsm()
return ecx
}
// LogicalCPU will return the Logical CPU the code is currently executing on.
// This is likely to change when the OS re-schedules the running thread
// to another CPU.
// If the current core cannot be detected, -1 will be returned.
func (c cpuInfo) logicalcpu() int {
if c.maxFunc < 1 {
return -1
}
_, ebx, _, _ := cpuid(1)
return int(ebx >> 24)
}
// VM Will return true if the cpu id indicates we are in
// a virtual machine. This is only a hint, and will very likely
// have many false negatives.
func (c cpuInfo) vm() bool {
switch c.vendorid {
case msvm, kvm, vmware, xenhvm:
return true
}
return false
}
// Flags contains detected cpu features and caracteristics
type flags uint64
// String returns a string representation of the detected
// CPU features.
func (f flags) String() string {
return strings.Join(f.strings(), ",")
}
// Strings returns and array of the detected features.
func (f flags) strings() []string {
s := support()
r := make([]string, 0, 20)
for i := uint(0); i < 64; i++ {
key := flags(1 << i)
val := flagNames[key]
if s&key != 0 {
r = append(r, val)
}
}
return r
}
func maxExtendedFunction() uint32 {
eax, _, _, _ := cpuid(0x80000000)
return eax
}
func maxFunctionID() uint32 {
a, _, _, _ := cpuid(0)
return a
}
func brandName() string {
if maxExtendedFunction() >= 0x80000004 {
v := make([]uint32, 0, 48)
for i := uint32(0); i < 3; i++ {
a, b, c, d := cpuid(0x80000002 + i)
v = append(v, a, b, c, d)
}
return strings.Trim(string(valAsString(v...)), " ")
}
return "unknown"
}
func threadsPerCore() int {
mfi := maxFunctionID()
if mfi < 0x4 || vendorID() != intel {
return 1
}
if mfi < 0xb {
_, b, _, d := cpuid(1)
if (d & (1 << 28)) != 0 {
// v will contain logical core count
v := (b >> 16) & 255
if v > 1 {
a4, _, _, _ := cpuid(4)
// physical cores
v2 := (a4 >> 26) + 1
if v2 > 0 {
return int(v) / int(v2)
}
}
}
return 1
}
_, b, _, _ := cpuidex(0xb, 0)
if b&0xffff == 0 {
return 1
}
return int(b & 0xffff)
}
func logicalCores() int {
mfi := maxFunctionID()
switch vendorID() {
case intel:
// Use this on old Intel processors
if mfi < 0xb {
if mfi < 1 {
return 0
}
// CPUID.1:EBX[23:16] represents the maximum number of addressable IDs (initial APIC ID)
// that can be assigned to logical processors in a physical package.
// The value may not be the same as the number of logical processors that are present in the hardware of a physical package.
_, ebx, _, _ := cpuid(1)
logical := (ebx >> 16) & 0xff
return int(logical)
}
_, b, _, _ := cpuidex(0xb, 1)
return int(b & 0xffff)
case amd:
_, b, _, _ := cpuid(1)
return int((b >> 16) & 0xff)
default:
return 0
}
}
func familyModel() (int, int) {
if maxFunctionID() < 0x1 {
return 0, 0
}
eax, _, _, _ := cpuid(1)
family := ((eax >> 8) & 0xf) + ((eax >> 20) & 0xff)
model := ((eax >> 4) & 0xf) + ((eax >> 12) & 0xf0)
return int(family), int(model)
}
func physicalCores() int {
switch vendorID() {
case intel:
return logicalCores() / threadsPerCore()
case amd:
if maxExtendedFunction() >= 0x80000008 {
_, _, c, _ := cpuid(0x80000008)
return int(c&0xff) + 1
}
}
return 0
}
// Except from http://en.wikipedia.org/wiki/CPUID#EAX.3D0:_Get_vendor_ID
var vendorMapping = map[string]vendor{
"AMDisbetter!": amd,
"AuthenticAMD": amd,
"CentaurHauls": via,
"GenuineIntel": intel,
"TransmetaCPU": transmeta,
"GenuineTMx86": transmeta,
"Geode by NSC": nsc,
"VIA VIA VIA ": via,
"KVMKVMKVMKVM": kvm,
"Microsoft Hv": msvm,
"VMwareVMware": vmware,
"XenVMMXenVMM": xenhvm,
}
func vendorID() vendor {
_, b, c, d := cpuid(0)
v := valAsString(b, d, c)
vend, ok := vendorMapping[string(v)]
if !ok {
return other
}
return vend
}
func cacheLine() int {
if maxFunctionID() < 0x1 {
return 0
}
_, ebx, _, _ := cpuid(1)
cache := (ebx & 0xff00) >> 5 // cflush size
if cache == 0 && maxExtendedFunction() >= 0x80000006 {
_, _, ecx, _ := cpuid(0x80000006)
cache = ecx & 0xff // cacheline size
}
// TODO: Read from Cache and TLB Information
return int(cache)
}
func (c *cpuInfo) cacheSize() {
c.cache.l1d = -1
c.cache.l1i = -1
c.cache.l2 = -1
c.cache.l3 = -1
vendor := vendorID()
switch vendor {
case intel:
if maxFunctionID() < 4 {
return
}
for i := uint32(0); ; i++ {
eax, ebx, ecx, _ := cpuidex(4, i)
cacheType := eax & 15
if cacheType == 0 {
break
}
cacheLevel := (eax >> 5) & 7
coherency := int(ebx&0xfff) + 1
partitions := int((ebx>>12)&0x3ff) + 1
associativity := int((ebx>>22)&0x3ff) + 1
sets := int(ecx) + 1
size := associativity * partitions * coherency * sets
switch cacheLevel {
case 1:
if cacheType == 1 {
// 1 = Data Cache
c.cache.l1d = size
} else if cacheType == 2 {
// 2 = Instruction Cache
c.cache.l1i = size
} else {
if c.cache.l1d < 0 {
c.cache.l1i = size
}
if c.cache.l1i < 0 {
c.cache.l1i = size
}
}
case 2:
c.cache.l2 = size
case 3:
c.cache.l3 = size
}
}
case amd:
// Untested.
if maxExtendedFunction() < 0x80000005 {
return
}
_, _, ecx, edx := cpuid(0x80000005)
c.cache.l1d = int(((ecx >> 24) & 0xFF) * 1024)
c.cache.l1i = int(((edx >> 24) & 0xFF) * 1024)
if maxExtendedFunction() < 0x80000006 {
return
}
_, _, ecx, _ = cpuid(0x80000006)
c.cache.l2 = int(((ecx >> 16) & 0xFFFF) * 1024)
}
return
}
func support() flags {
mfi := maxFunctionID()
vend := vendorID()
if mfi < 0x1 {
return 0
}
rval := uint64(0)
_, _, c, d := cpuid(1)
if (d & (1 << 15)) != 0 {
rval |= cmov
}
if (d & (1 << 23)) != 0 {
rval |= mmx
}
if (d & (1 << 25)) != 0 {
rval |= mmxext
}
if (d & (1 << 25)) != 0 {
rval |= sse
}
if (d & (1 << 26)) != 0 {
rval |= sse2
}
if (c & 1) != 0 {
rval |= sse3
}
if (c & 0x00000200) != 0 {
rval |= ssse3
}
if (c & 0x00080000) != 0 {
rval |= sse4
}
if (c & 0x00100000) != 0 {
rval |= sse42
}
if (c & (1 << 25)) != 0 {
rval |= aesni
}
if (c & (1 << 1)) != 0 {
rval |= clmul
}
if c&(1<<23) != 0 {
rval |= popcnt
}
if c&(1<<30) != 0 {
rval |= rdrand
}
if c&(1<<29) != 0 {
rval |= f16c
}
if c&(1<<13) != 0 {
rval |= cx16
}
if vend == intel && (d&(1<<28)) != 0 && mfi >= 4 {
if threadsPerCore() > 1 {
rval |= htt
}
}
// Check XGETBV, OXSAVE and AVX bits
if c&(1<<26) != 0 && c&(1<<27) != 0 && c&(1<<28) != 0 {
// Check for OS support
eax, _ := xgetbv(0)
if (eax & 0x6) == 0x6 {
rval |= avx
if (c & 0x00001000) != 0 {
rval |= fma3
}
}
}
// Check AVX2, AVX2 requires OS support, but BMI1/2 don't.
if mfi >= 7 {
_, ebx, ecx, _ := cpuidex(7, 0)
if (rval&avx) != 0 && (ebx&0x00000020) != 0 {
rval |= avx2
}
if (ebx & 0x00000008) != 0 {
rval |= bmi1
if (ebx & 0x00000100) != 0 {
rval |= bmi2
}
}
if ebx&(1<<4) != 0 {
rval |= hle
}
if ebx&(1<<9) != 0 {
rval |= erms
}
if ebx&(1<<11) != 0 {
rval |= rtm
}
if ebx&(1<<14) != 0 {
rval |= mpx
}
if ebx&(1<<18) != 0 {
rval |= rdseed
}
if ebx&(1<<19) != 0 {
rval |= adx
}
if ebx&(1<<29) != 0 {
rval |= sha
}
// Only detect AVX-512 features if XGETBV is supported
if c&((1<<26)|(1<<27)) == (1<<26)|(1<<27) {
// Check for OS support
eax, _ := xgetbv(0)
// Verify that XCR0[7:5] = ‘111b’ (OPMASK state, upper 256-bit of ZMM0-ZMM15 and
// ZMM16-ZMM31 state are enabled by OS)
/// and that XCR0[2:1] = ‘11b’ (XMM state and YMM state are enabled by OS).
if (eax>>5)&7 == 7 && (eax>>1)&3 == 3 {
if ebx&(1<<16) != 0 {
rval |= avx512f
}
if ebx&(1<<17) != 0 {
rval |= avx512dq
}
if ebx&(1<<21) != 0 {
rval |= avx512ifma
}
if ebx&(1<<26) != 0 {
rval |= avx512pf
}
if ebx&(1<<27) != 0 {
rval |= avx512er
}
if ebx&(1<<28) != 0 {
rval |= avx512cd
}
if ebx&(1<<30) != 0 {
rval |= avx512bw
}
if ebx&(1<<31) != 0 {
rval |= avx512vl
}
// ecx
if ecx&(1<<1) != 0 {
rval |= avx512vbmi
}
}
}
}
if maxExtendedFunction() >= 0x80000001 {
_, _, c, d := cpuid(0x80000001)
if (c & (1 << 5)) != 0 {
rval |= lzcnt
rval |= popcnt
}
if (d & (1 << 31)) != 0 {
rval |= amd3dnow
}
if (d & (1 << 30)) != 0 {
rval |= amd3dnowext
}
if (d & (1 << 23)) != 0 {
rval |= mmx
}
if (d & (1 << 22)) != 0 {
rval |= mmxext
}
if (c & (1 << 6)) != 0 {
rval |= sse4a
}
if d&(1<<20) != 0 {
rval |= nx
}
if d&(1<<27) != 0 {
rval |= rdtscp
}
/* Allow for selectively disabling SSE2 functions on AMD processors
with SSE2 support but not SSE4a. This includes Athlon64, some
Opteron, and some Sempron processors. MMX, SSE, or 3DNow! are faster
than SSE2 often enough to utilize this special-case flag.
AV_CPU_FLAG_SSE2 and AV_CPU_FLAG_SSE2SLOW are both set in this case
so that SSE2 is used unless explicitly disabled by checking
AV_CPU_FLAG_SSE2SLOW. */
if vendorID() != intel &&
rval&sse2 != 0 && (c&0x00000040) == 0 {
rval |= sse2slow
}
/* XOP and FMA4 use the AVX instruction coding scheme, so they can't be
* used unless the OS has AVX support. */
if (rval & avx) != 0 {
if (c & 0x00000800) != 0 {
rval |= xop
}
if (c & 0x00010000) != 0 {
rval |= fma4
}
}
if vendorID() == intel {
family, model := familyModel()
if family == 6 && (model == 9 || model == 13 || model == 14) {
/* 6/9 (pentium-m "banias"), 6/13 (pentium-m "dothan"), and
* 6/14 (core1 "yonah") theoretically support sse2, but it's
* usually slower than mmx. */
if (rval & sse2) != 0 {
rval |= sse2slow
}
if (rval & sse3) != 0 {
rval |= sse3slow
}
}
/* The Atom processor has SSSE3 support, which is useful in many cases,
* but sometimes the SSSE3 version is slower than the SSE2 equivalent
* on the Atom, but is generally faster on other processors supporting
* SSSE3. This flag allows for selectively disabling certain SSSE3
* functions on the Atom. */
if family == 6 && model == 28 {
rval |= atom
}
}
}
return flags(rval)
}
func valAsString(values ...uint32) []byte {
r := make([]byte, 4*len(values))
for i, v := range values {
dst := r[i*4:]
dst[0] = byte(v & 0xff)
dst[1] = byte((v >> 8) & 0xff)
dst[2] = byte((v >> 16) & 0xff)
dst[3] = byte((v >> 24) & 0xff)
switch {
case dst[0] == 0:
return r[:i*4]
case dst[1] == 0:
return r[:i*4+1]
case dst[2] == 0:
return r[:i*4+2]
case dst[3] == 0:
return r[:i*4+3]
}
}
return r
}

42
vendor/github.com/klauspost/cpuid/private/cpuid_386.s generated vendored
View File

@@ -1,42 +0,0 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORL CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+4(FP)
MOVL BX, ebx+8(FP)
MOVL CX, ecx+12(FP)
MOVL DX, edx+16(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func xgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+4(FP)
MOVL DX, edx+8(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET

42
vendor/github.com/klauspost/cpuid/private/cpuid_amd64.s generated vendored
View File

@@ -1,42 +0,0 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
//+build amd64,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORQ CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmXgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+8(FP)
MOVL DX, edx+12(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET

17
vendor/github.com/klauspost/cpuid/private/cpuid_detect_intel.go generated vendored
View File

@@ -1,17 +0,0 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo amd64,!gccgo
package cpuid
func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
func asmXgetbv(index uint32) (eax, edx uint32)
func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
func initCPU() {
cpuid = asmCpuid
cpuidex = asmCpuidex
xgetbv = asmXgetbv
rdtscpAsm = asmRdtscpAsm
}

23
vendor/github.com/klauspost/cpuid/private/cpuid_detect_ref.go generated vendored
View File

@@ -1,23 +0,0 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build !amd64,!386 gccgo
package cpuid
func initCPU() {
cpuid = func(op uint32) (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
cpuidex = func(op, op2 uint32) (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
xgetbv = func(index uint32) (eax, edx uint32) {
return 0, 0
}
rdtscpAsm = func() (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
}

719
vendor/github.com/klauspost/cpuid/private/cpuid_test.go generated vendored
View File

@@ -1,719 +0,0 @@
// Generated, DO NOT EDIT,
// but copy it to your own project and rename the package.
// See more at http://github.com/klauspost/cpuid
package cpuid
import (
"fmt"
"testing"
)
// There is no real way to test a CPU identifier, since results will
// obviously differ on each machine.
func TestCPUID(t *testing.T) {
n := maxFunctionID()
t.Logf("Max Function:0x%x\n", n)
n = maxExtendedFunction()
t.Logf("Max Extended Function:0x%x\n", n)
t.Log("Name:", cpu.brandname)
t.Log("PhysicalCores:", cpu.physicalcores)
t.Log("ThreadsPerCore:", cpu.threadspercore)
t.Log("LogicalCores:", cpu.logicalcores)
t.Log("Family", cpu.family, "Model:", cpu.model)
t.Log("Features:", cpu.features)
t.Log("Cacheline bytes:", cpu.cacheline)
t.Log("L1 Instruction Cache:", cpu.cache.l1i, "bytes")
t.Log("L1 Data Cache:", cpu.cache.l1d, "bytes")
t.Log("L2 Cache:", cpu.cache.l2, "bytes")
t.Log("L3 Cache:", cpu.cache.l3, "bytes")
if cpu.sse2() {
t.Log("We have SSE2")
}
}
func TestDumpCPUID(t *testing.T) {
n := int(maxFunctionID())
for i := 0; i <= n; i++ {
a, b, c, d := cpuidex(uint32(i), 0)
t.Logf("CPUID %08x: %08x-%08x-%08x-%08x", i, a, b, c, d)
ex := uint32(1)
for {
a2, b2, c2, d2 := cpuidex(uint32(i), ex)
if a2 == a && b2 == b && d2 == d || ex > 50 || a2 == 0 {
break
}
t.Logf("CPUID %08x: %08x-%08x-%08x-%08x", i, a2, b2, c2, d2)
a, b, c, d = a2, b2, c2, d2
ex++
}
}
n2 := maxExtendedFunction()
for i := uint32(0x80000000); i <= n2; i++ {
a, b, c, d := cpuid(i)
t.Logf("CPUID %08x: %08x-%08x-%08x-%08x", i, a, b, c, d)
}
}
func example() {
// Print basic CPU information:
fmt.Println("Name:", cpu.brandname)
fmt.Println("PhysicalCores:", cpu.physicalcores)
fmt.Println("ThreadsPerCore:", cpu.threadspercore)
fmt.Println("LogicalCores:", cpu.logicalcores)
fmt.Println("Family", cpu.family, "Model:", cpu.model)
fmt.Println("Features:", cpu.features)
fmt.Println("Cacheline bytes:", cpu.cacheline)
// Test if we have a specific feature:
if cpu.sse() {
fmt.Println("We have Streaming SIMD Extensions")
}
}
func TestBrandNameZero(t *testing.T) {
if len(cpu.brandname) > 0 {
// Cut out last byte
last := []byte(cpu.brandname[len(cpu.brandname)-1:])
if last[0] == 0 {
t.Fatal("last byte was zero")
} else if last[0] == 32 {
t.Fatal("whitespace wasn't trimmed")
}
}
}
// Generated here: http://play.golang.org/p/mko-0tFt0Q
// TestCmov tests Cmov() function
func TestCmov(t *testing.T) {
got := cpu.cmov()
expected := cpu.features&cmov == cmov
if got != expected {
t.Fatalf("Cmov: expected %v, got %v", expected, got)
}
t.Log("CMOV Support:", got)
}
// TestAmd3dnow tests Amd3dnow() function
func TestAmd3dnow(t *testing.T) {
got := cpu.amd3dnow()
expected := cpu.features&amd3dnow == amd3dnow
if got != expected {
t.Fatalf("Amd3dnow: expected %v, got %v", expected, got)
}
t.Log("AMD3DNOW Support:", got)
}
// TestAmd3dnowExt tests Amd3dnowExt() function
func TestAmd3dnowExt(t *testing.T) {
got := cpu.amd3dnowext()
expected := cpu.features&amd3dnowext == amd3dnowext
if got != expected {
t.Fatalf("Amd3dnowExt: expected %v, got %v", expected, got)
}
t.Log("AMD3DNOWEXT Support:", got)
}
// TestMMX tests MMX() function
func TestMMX(t *testing.T) {
got := cpu.mmx()
expected := cpu.features&mmx == mmx
if got != expected {
t.Fatalf("MMX: expected %v, got %v", expected, got)
}
t.Log("MMX Support:", got)
}
// TestMMXext tests MMXext() function
func TestMMXext(t *testing.T) {
got := cpu.mmxext()
expected := cpu.features&mmxext == mmxext
if got != expected {
t.Fatalf("MMXExt: expected %v, got %v", expected, got)
}
t.Log("MMXEXT Support:", got)
}
// TestSSE tests SSE() function
func TestSSE(t *testing.T) {
got := cpu.sse()
expected := cpu.features&sse == sse
if got != expected {
t.Fatalf("SSE: expected %v, got %v", expected, got)
}
t.Log("SSE Support:", got)
}
// TestSSE2 tests SSE2() function
func TestSSE2(t *testing.T) {
got := cpu.sse2()
expected := cpu.features&sse2 == sse2
if got != expected {
t.Fatalf("SSE2: expected %v, got %v", expected, got)
}
t.Log("SSE2 Support:", got)
}
// TestSSE3 tests SSE3() function
func TestSSE3(t *testing.T) {
got := cpu.sse3()
expected := cpu.features&sse3 == sse3
if got != expected {
t.Fatalf("SSE3: expected %v, got %v", expected, got)
}
t.Log("SSE3 Support:", got)
}
// TestSSSE3 tests SSSE3() function
func TestSSSE3(t *testing.T) {
got := cpu.ssse3()
expected := cpu.features&ssse3 == ssse3
if got != expected {
t.Fatalf("SSSE3: expected %v, got %v", expected, got)
}
t.Log("SSSE3 Support:", got)
}
// TestSSE4 tests SSE4() function
func TestSSE4(t *testing.T) {
got := cpu.sse4()
expected := cpu.features&sse4 == sse4
if got != expected {
t.Fatalf("SSE4: expected %v, got %v", expected, got)
}
t.Log("SSE4 Support:", got)
}
// TestSSE42 tests SSE42() function
func TestSSE42(t *testing.T) {
got := cpu.sse42()
expected := cpu.features&sse42 == sse42
if got != expected {
t.Fatalf("SSE42: expected %v, got %v", expected, got)
}
t.Log("SSE42 Support:", got)
}
// TestAVX tests AVX() function
func TestAVX(t *testing.T) {
got := cpu.avx()
expected := cpu.features&avx == avx
if got != expected {
t.Fatalf("AVX: expected %v, got %v", expected, got)
}
t.Log("AVX Support:", got)
}
// TestAVX2 tests AVX2() function
func TestAVX2(t *testing.T) {
got := cpu.avx2()
expected := cpu.features&avx2 == avx2
if got != expected {
t.Fatalf("AVX2: expected %v, got %v", expected, got)
}
t.Log("AVX2 Support:", got)
}
// TestFMA3 tests FMA3() function
func TestFMA3(t *testing.T) {
got := cpu.fma3()
expected := cpu.features&fma3 == fma3
if got != expected {
t.Fatalf("FMA3: expected %v, got %v", expected, got)
}
t.Log("FMA3 Support:", got)
}
// TestFMA4 tests FMA4() function
func TestFMA4(t *testing.T) {
got := cpu.fma4()
expected := cpu.features&fma4 == fma4
if got != expected {
t.Fatalf("FMA4: expected %v, got %v", expected, got)
}
t.Log("FMA4 Support:", got)
}
// TestXOP tests XOP() function
func TestXOP(t *testing.T) {
got := cpu.xop()
expected := cpu.features&xop == xop
if got != expected {
t.Fatalf("XOP: expected %v, got %v", expected, got)
}
t.Log("XOP Support:", got)
}
// TestF16C tests F16C() function
func TestF16C(t *testing.T) {
got := cpu.f16c()
expected := cpu.features&f16c == f16c
if got != expected {
t.Fatalf("F16C: expected %v, got %v", expected, got)
}
t.Log("F16C Support:", got)
}
// TestCX16 tests CX16() function
func TestCX16(t *testing.T) {
got := cpu.cx16()
expected := cpu.features&cx16 == cx16
if got != expected {
t.Fatalf("CX16: expected %v, got %v", expected, got)
}
t.Log("CX16 Support:", got)
}
// TestBMI1 tests BMI1() function
func TestBMI1(t *testing.T) {
got := cpu.bmi1()
expected := cpu.features&bmi1 == bmi1
if got != expected {
t.Fatalf("BMI1: expected %v, got %v", expected, got)
}
t.Log("BMI1 Support:", got)
}
// TestBMI2 tests BMI2() function
func TestBMI2(t *testing.T) {
got := cpu.bmi2()
expected := cpu.features&bmi2 == bmi2
if got != expected {
t.Fatalf("BMI2: expected %v, got %v", expected, got)
}
t.Log("BMI2 Support:", got)
}
// TestTBM tests TBM() function
func TestTBM(t *testing.T) {
got := cpu.tbm()
expected := cpu.features&tbm == tbm
if got != expected {
t.Fatalf("TBM: expected %v, got %v", expected, got)
}
t.Log("TBM Support:", got)
}
// TestLzcnt tests Lzcnt() function
func TestLzcnt(t *testing.T) {
got := cpu.lzcnt()
expected := cpu.features&lzcnt == lzcnt
if got != expected {
t.Fatalf("Lzcnt: expected %v, got %v", expected, got)
}
t.Log("LZCNT Support:", got)
}
// TestLzcnt tests Lzcnt() function
func TestPopcnt(t *testing.T) {
got := cpu.popcnt()
expected := cpu.features&popcnt == popcnt
if got != expected {
t.Fatalf("Popcnt: expected %v, got %v", expected, got)
}
t.Log("POPCNT Support:", got)
}
// TestAesNi tests AesNi() function
func TestAesNi(t *testing.T) {
got := cpu.aesni()
expected := cpu.features&aesni == aesni
if got != expected {
t.Fatalf("AesNi: expected %v, got %v", expected, got)
}
t.Log("AESNI Support:", got)
}
// TestHTT tests HTT() function
func TestHTT(t *testing.T) {
got := cpu.htt()
expected := cpu.features&htt == htt
if got != expected {
t.Fatalf("HTT: expected %v, got %v", expected, got)
}
t.Log("HTT Support:", got)
}
// TestClmul tests Clmul() function
func TestClmul(t *testing.T) {
got := cpu.clmul()
expected := cpu.features&clmul == clmul
if got != expected {
t.Fatalf("Clmul: expected %v, got %v", expected, got)
}
t.Log("CLMUL Support:", got)
}
// TestSSE2Slow tests SSE2Slow() function
func TestSSE2Slow(t *testing.T) {
got := cpu.sse2slow()
expected := cpu.features&sse2slow == sse2slow
if got != expected {
t.Fatalf("SSE2Slow: expected %v, got %v", expected, got)
}
t.Log("SSE2SLOW Support:", got)
}
// TestSSE3Slow tests SSE3slow() function
func TestSSE3Slow(t *testing.T) {
got := cpu.sse3slow()
expected := cpu.features&sse3slow == sse3slow
if got != expected {
t.Fatalf("SSE3slow: expected %v, got %v", expected, got)
}
t.Log("SSE3SLOW Support:", got)
}
// TestAtom tests Atom() function
func TestAtom(t *testing.T) {
got := cpu.atom()
expected := cpu.features&atom == atom
if got != expected {
t.Fatalf("Atom: expected %v, got %v", expected, got)
}
t.Log("ATOM Support:", got)
}
// TestNX tests NX() function (NX (No-Execute) bit)
func TestNX(t *testing.T) {
got := cpu.nx()
expected := cpu.features&nx == nx
if got != expected {
t.Fatalf("NX: expected %v, got %v", expected, got)
}
t.Log("NX Support:", got)
}
// TestSSE4A tests SSE4A() function (AMD Barcelona microarchitecture SSE4a instructions)
func TestSSE4A(t *testing.T) {
got := cpu.sse4a()
expected := cpu.features&sse4a == sse4a
if got != expected {
t.Fatalf("SSE4A: expected %v, got %v", expected, got)
}
t.Log("SSE4A Support:", got)
}
// TestHLE tests HLE() function (Hardware Lock Elision)
func TestHLE(t *testing.T) {
got := cpu.hle()
expected := cpu.features&hle == hle
if got != expected {
t.Fatalf("HLE: expected %v, got %v", expected, got)
}
t.Log("HLE Support:", got)
}
// TestRTM tests RTM() function (Restricted Transactional Memory)
func TestRTM(t *testing.T) {
got := cpu.rtm()
expected := cpu.features&rtm == rtm
if got != expected {
t.Fatalf("RTM: expected %v, got %v", expected, got)
}
t.Log("RTM Support:", got)
}
// TestRdrand tests RDRAND() function (RDRAND instruction is available)
func TestRdrand(t *testing.T) {
got := cpu.rdrand()
expected := cpu.features&rdrand == rdrand
if got != expected {
t.Fatalf("Rdrand: expected %v, got %v", expected, got)
}
t.Log("Rdrand Support:", got)
}
// TestRdseed tests RDSEED() function (RDSEED instruction is available)
func TestRdseed(t *testing.T) {
got := cpu.rdseed()
expected := cpu.features&rdseed == rdseed
if got != expected {
t.Fatalf("Rdseed: expected %v, got %v", expected, got)
}
t.Log("Rdseed Support:", got)
}
// TestADX tests ADX() function (Intel ADX (Multi-Precision Add-Carry Instruction Extensions))
func TestADX(t *testing.T) {
got := cpu.adx()
expected := cpu.features&adx == adx
if got != expected {
t.Fatalf("ADX: expected %v, got %v", expected, got)
}
t.Log("ADX Support:", got)
}
// TestSHA tests SHA() function (Intel SHA Extensions)
func TestSHA(t *testing.T) {
got := cpu.sha()
expected := cpu.features&sha == sha
if got != expected {
t.Fatalf("SHA: expected %v, got %v", expected, got)
}
t.Log("SHA Support:", got)
}
// TestAVX512F tests AVX512F() function (AVX-512 Foundation)
func TestAVX512F(t *testing.T) {
got := cpu.avx512f()
expected := cpu.features&avx512f == avx512f
if got != expected {
t.Fatalf("AVX512F: expected %v, got %v", expected, got)
}
t.Log("AVX512F Support:", got)
}
// TestAVX512DQ tests AVX512DQ() function (AVX-512 Doubleword and Quadword Instructions)
func TestAVX512DQ(t *testing.T) {
got := cpu.avx512dq()
expected := cpu.features&avx512dq == avx512dq
if got != expected {
t.Fatalf("AVX512DQ: expected %v, got %v", expected, got)
}
t.Log("AVX512DQ Support:", got)
}
// TestAVX512IFMA tests AVX512IFMA() function (AVX-512 Integer Fused Multiply-Add Instructions)
func TestAVX512IFMA(t *testing.T) {
got := cpu.avx512ifma()
expected := cpu.features&avx512ifma == avx512ifma
if got != expected {
t.Fatalf("AVX512IFMA: expected %v, got %v", expected, got)
}
t.Log("AVX512IFMA Support:", got)
}
// TestAVX512PF tests AVX512PF() function (AVX-512 Prefetch Instructions)
func TestAVX512PF(t *testing.T) {
got := cpu.avx512pf()
expected := cpu.features&avx512pf == avx512pf
if got != expected {
t.Fatalf("AVX512PF: expected %v, got %v", expected, got)
}
t.Log("AVX512PF Support:", got)
}
// TestAVX512ER tests AVX512ER() function (AVX-512 Exponential and Reciprocal Instructions)
func TestAVX512ER(t *testing.T) {
got := cpu.avx512er()
expected := cpu.features&avx512er == avx512er
if got != expected {
t.Fatalf("AVX512ER: expected %v, got %v", expected, got)
}
t.Log("AVX512ER Support:", got)
}
// TestAVX512CD tests AVX512CD() function (AVX-512 Conflict Detection Instructions)
func TestAVX512CD(t *testing.T) {
got := cpu.avx512cd()
expected := cpu.features&avx512cd == avx512cd
if got != expected {
t.Fatalf("AVX512CD: expected %v, got %v", expected, got)
}
t.Log("AVX512CD Support:", got)
}
// TestAVX512BW tests AVX512BW() function (AVX-512 Byte and Word Instructions)
func TestAVX512BW(t *testing.T) {
got := cpu.avx512bw()
expected := cpu.features&avx512bw == avx512bw
if got != expected {
t.Fatalf("AVX512BW: expected %v, got %v", expected, got)
}
t.Log("AVX512BW Support:", got)
}
// TestAVX512VL tests AVX512VL() function (AVX-512 Vector Length Extensions)
func TestAVX512VL(t *testing.T) {
got := cpu.avx512vl()
expected := cpu.features&avx512vl == avx512vl
if got != expected {
t.Fatalf("AVX512VL: expected %v, got %v", expected, got)
}
t.Log("AVX512VL Support:", got)
}
// TestAVX512VL tests AVX512VBMI() function (AVX-512 Vector Bit Manipulation Instructions)
func TestAVX512VBMI(t *testing.T) {
got := cpu.avx512vbmi()
expected := cpu.features&avx512vbmi == avx512vbmi
if got != expected {
t.Fatalf("AVX512VBMI: expected %v, got %v", expected, got)
}
t.Log("AVX512VBMI Support:", got)
}
// TestMPX tests MPX() function (Intel MPX (Memory Protection Extensions))
func TestMPX(t *testing.T) {
got := cpu.mpx()
expected := cpu.features&mpx == mpx
if got != expected {
t.Fatalf("MPX: expected %v, got %v", expected, got)
}
t.Log("MPX Support:", got)
}
// TestERMS tests ERMS() function (Enhanced REP MOVSB/STOSB)
func TestERMS(t *testing.T) {
got := cpu.erms()
expected := cpu.features&erms == erms
if got != expected {
t.Fatalf("ERMS: expected %v, got %v", expected, got)
}
t.Log("ERMS Support:", got)
}
// TestVendor writes the detected vendor. Will be 0 if unknown
func TestVendor(t *testing.T) {
t.Log("Vendor ID:", cpu.vendorid)
}
// Intel returns true if vendor is recognized as Intel
func TestIntel(t *testing.T) {
got := cpu.intel()
expected := cpu.vendorid == intel
if got != expected {
t.Fatalf("TestIntel: expected %v, got %v", expected, got)
}
t.Log("TestIntel:", got)
}
// AMD returns true if vendor is recognized as AMD
func TestAMD(t *testing.T) {
got := cpu.amd()
expected := cpu.vendorid == amd
if got != expected {
t.Fatalf("TestAMD: expected %v, got %v", expected, got)
}
t.Log("TestAMD:", got)
}
// Transmeta returns true if vendor is recognized as Transmeta
func TestTransmeta(t *testing.T) {
got := cpu.transmeta()
expected := cpu.vendorid == transmeta
if got != expected {
t.Fatalf("TestTransmeta: expected %v, got %v", expected, got)
}
t.Log("TestTransmeta:", got)
}
// NSC returns true if vendor is recognized as National Semiconductor
func TestNSC(t *testing.T) {
got := cpu.nsc()
expected := cpu.vendorid == nsc
if got != expected {
t.Fatalf("TestNSC: expected %v, got %v", expected, got)
}
t.Log("TestNSC:", got)
}
// VIA returns true if vendor is recognized as VIA
func TestVIA(t *testing.T) {
got := cpu.via()
expected := cpu.vendorid == via
if got != expected {
t.Fatalf("TestVIA: expected %v, got %v", expected, got)
}
t.Log("TestVIA:", got)
}
// Test VM function
func TestVM(t *testing.T) {
t.Log("Vendor ID:", cpu.vm())
}
// Test RTCounter function
func TestRtCounter(t *testing.T) {
a := cpu.rtcounter()
b := cpu.rtcounter()
t.Log("CPU Counter:", a, b, b-a)
}
// Prints the value of Ia32TscAux()
func TestIa32TscAux(t *testing.T) {
ecx := cpu.ia32tscaux()
t.Logf("Ia32TscAux:0x%x\n", ecx)
if ecx != 0 {
chip := (ecx & 0xFFF000) >> 12
core := ecx & 0xFFF
t.Log("Likely chip, core:", chip, core)
}
}
func TestThreadsPerCoreNZ(t *testing.T) {
if cpu.threadspercore == 0 {
t.Fatal("threads per core is zero")
}
}
// Prints the value of LogicalCPU()
func TestLogicalCPU(t *testing.T) {
t.Log("Currently executing on cpu:", cpu.logicalcpu())
}
func TestMaxFunction(t *testing.T) {
expect := maxFunctionID()
if cpu.maxFunc != expect {
t.Fatal("Max function does not match, expected", expect, "but got", cpu.maxFunc)
}
expect = maxExtendedFunction()
if cpu.maxExFunc != expect {
t.Fatal("Max Extended function does not match, expected", expect, "but got", cpu.maxFunc)
}
}
// This example will calculate the chip/core number on Linux
// Linux encodes numa id (<<12) and core id (8bit) into TSC_AUX.
func examplecpuinfo_ia32tscaux(t *testing.T) {
ecx := cpu.ia32tscaux()
if ecx == 0 {
fmt.Println("Unknown CPU ID")
return
}
chip := (ecx & 0xFFF000) >> 12
core := ecx & 0xFFF
fmt.Println("Chip, Core:", chip, core)
}
/*
func TestPhysical(t *testing.T) {
var test16 = "CPUID 00000000: 0000000d-756e6547-6c65746e-49656e69 \nCPUID 00000001: 000206d7-03200800-1fbee3ff-bfebfbff \nCPUID 00000002: 76035a01-00f0b2ff-00000000-00ca0000 \nCPUID 00000003: 00000000-00000000-00000000-00000000 \nCPUID 00000004: 3c004121-01c0003f-0000003f-00000000 \nCPUID 00000004: 3c004122-01c0003f-0000003f-00000000 \nCPUID 00000004: 3c004143-01c0003f-000001ff-00000000 \nCPUID 00000004: 3c07c163-04c0003f-00003fff-00000006 \nCPUID 00000005: 00000040-00000040-00000003-00021120 \nCPUID 00000006: 00000075-00000002-00000009-00000000 \nCPUID 00000007: 00000000-00000000-00000000-00000000 \nCPUID 00000008: 00000000-00000000-00000000-00000000 \nCPUID 00000009: 00000001-00000000-00000000-00000000 \nCPUID 0000000a: 07300403-00000000-00000000-00000603 \nCPUID 0000000b: 00000000-00000000-00000003-00000003 \nCPUID 0000000b: 00000005-00000010-00000201-00000003 \nCPUID 0000000c: 00000000-00000000-00000000-00000000 \nCPUID 0000000d: 00000007-00000340-00000340-00000000 \nCPUID 0000000d: 00000001-00000000-00000000-00000000 \nCPUID 0000000d: 00000100-00000240-00000000-00000000 \nCPUID 80000000: 80000008-00000000-00000000-00000000 \nCPUID 80000001: 00000000-00000000-00000001-2c100800 \nCPUID 80000002: 20202020-49202020-6c65746e-20295228 \nCPUID 80000003: 6e6f6558-20295228-20555043-322d3545 \nCPUID 80000004: 20303636-20402030-30322e32-007a4847 \nCPUID 80000005: 00000000-00000000-00000000-00000000 \nCPUID 80000006: 00000000-00000000-01006040-00000000 \nCPUID 80000007: 00000000-00000000-00000000-00000100 \nCPUID 80000008: 0000302e-00000000-00000000-00000000"
restore := mockCPU([]byte(test16))
Detect()
t.Log("Name:", CPU.BrandName)
n := maxFunctionID()
t.Logf("Max Function:0x%x\n", n)
n = maxExtendedFunction()
t.Logf("Max Extended Function:0x%x\n", n)
t.Log("PhysicalCores:", CPU.PhysicalCores)
t.Log("ThreadsPerCore:", CPU.ThreadsPerCore)
t.Log("LogicalCores:", CPU.LogicalCores)
t.Log("Family", CPU.Family, "Model:", CPU.Model)
t.Log("Features:", CPU.Features)
t.Log("Cacheline bytes:", CPU.CacheLine)
t.Log("L1 Instruction Cache:", CPU.Cache.L1I, "bytes")
t.Log("L1 Data Cache:", CPU.Cache.L1D, "bytes")
t.Log("L2 Cache:", CPU.Cache.L2, "bytes")
t.Log("L3 Cache:", CPU.Cache.L3, "bytes")
if CPU.LogicalCores > 0 && CPU.PhysicalCores > 0 {
if CPU.LogicalCores != CPU.PhysicalCores*CPU.ThreadsPerCore {
t.Fatalf("Core count mismatch, LogicalCores (%d) != PhysicalCores (%d) * CPU.ThreadsPerCore (%d)",
CPU.LogicalCores, CPU.PhysicalCores, CPU.ThreadsPerCore)
}
}
if CPU.ThreadsPerCore > 1 && !CPU.HTT() {
t.Fatalf("Hyperthreading not detected")
}
if CPU.ThreadsPerCore == 1 && CPU.HTT() {
t.Fatalf("Hyperthreading detected, but only 1 Thread per core")
}
restore()
Detect()
TestCPUID(t)
}
*/

BIN
vendor/github.com/klauspost/cpuid/testdata/cpuid_data.zip generated vendored
View File

Binary file not shown.

77
vendor/github.com/klauspost/cpuid/testdata/getall.go generated vendored
View File

@@ -1,77 +0,0 @@
package main
import (
"archive/zip"
_ "bytes"
"fmt"
"golang.org/x/net/html"
"io"
"net/http"
"os"
"strings"
)
// Download all CPUID dumps from http://users.atw.hu/instlatx64/
func main() {
resp, err := http.Get("http://users.atw.hu/instlatx64/?")
if err != nil {
panic(err)
}
node, err := html.Parse(resp.Body)
if err != nil {
panic(err)
}
file, err := os.Create("cpuid_data.zip")
if err != nil {
panic(err)
}
defer file.Close()
gw := zip.NewWriter(file)
var f func(*html.Node)
f = func(n *html.Node) {
if n.Type == html.ElementNode && n.Data == "a" {
for _, a := range n.Attr {
if a.Key == "href" {
err := ParseURL(a.Val, gw)
if err != nil {
panic(err)
}
break
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
}
f(node)
err = gw.Close()
if err != nil {
panic(err)
}
}
func ParseURL(s string, gw *zip.Writer) error {
if strings.Contains(s, "CPUID.txt") {
fmt.Println("Adding", "http://users.atw.hu/instlatx64/"+s)
resp, err := http.Get("http://users.atw.hu/instlatx64/" + s)
if err != nil {
fmt.Println("Error getting ", s, ":", err)
}
defer resp.Body.Close()
w, err := gw.Create(s)
if err != nil {
return err
}
_, err = io.Copy(w, resp.Body)
if err != nil {
return err
}
}
return nil
}

24
vendor/github.com/klauspost/reedsolomon/.gitignore generated vendored
View File

@@ -1,24 +0,0 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
*.test
*.prof

33
vendor/github.com/klauspost/reedsolomon/.travis.yml generated vendored
View File

@@ -1,33 +0,0 @@
language: go
sudo: false
os:
- linux
- osx
go:
- 1.5
- 1.6
- 1.7
- 1.8
- master
install:
- go get ./...
script:
- go vet ./...
- go test -v -cpu=1,2,4 .
- go test -v -cpu=1,2,4 -short -race .
- go test -tags=noasm -v -cpu=1,2,4 -short -race .
- go build examples/simple-decoder.go
- go build examples/simple-encoder.go
- go build examples/stream-decoder.go
- go build examples/stream-encoder.go
- diff <(gofmt -d .) <("")
matrix:
allow_failures:
- go: 'master'
fast_finish: true

23
vendor/github.com/klauspost/reedsolomon/LICENSE generated vendored
View File

@@ -1,23 +0,0 @@
The MIT License (MIT)
Copyright (c) 2015 Klaus Post
Copyright (c) 2015 Backblaze
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

216
vendor/github.com/klauspost/reedsolomon/README.md generated vendored
View File

@@ -1,216 +0,0 @@
# Reed-Solomon
[![GoDoc][1]][2] [![Build Status][3]][4]
[1]: https://godoc.org/github.com/klauspost/reedsolomon?status.svg
[2]: https://godoc.org/github.com/klauspost/reedsolomon
[3]: https://travis-ci.org/klauspost/reedsolomon.svg?branch=master
[4]: https://travis-ci.org/klauspost/reedsolomon
Reed-Solomon Erasure Coding in Go, with speeds exceeding 1GB/s/cpu core implemented in pure Go.
This is a golang port of the [JavaReedSolomon](https://github.com/Backblaze/JavaReedSolomon) library released by [Backblaze](http://backblaze.com), with some additional optimizations.
For an introduction on erasure coding, see the post on the [Backblaze blog](https://www.backblaze.com/blog/reed-solomon/).
Package home: https://github.com/klauspost/reedsolomon
Godoc: https://godoc.org/github.com/klauspost/reedsolomon
# Installation
To get the package use the standard:
```bash
go get github.com/klauspost/reedsolomon
```
# Usage
This section assumes you know the basics of Reed-Solomon encoding. A good start is this [Backblaze blog post](https://www.backblaze.com/blog/reed-solomon/).
This package performs the calculation of the parity sets. The usage is therefore relatively simple.
First of all, you need to choose your distribution of data and parity shards. A 'good' distribution is very subjective, and will depend a lot on your usage scenario. A good starting point is above 5 and below 257 data shards (the maximum supported number), and the number of parity shards to be 2 or above, and below the number of data shards.
To create an encoder with 10 data shards (where your data goes) and 3 parity shards (calculated):
```Go
enc, err := reedsolomon.New(10, 3)
```
This encoder will work for all parity sets with this distribution of data and parity shards. The error will only be set if you specify 0 or negative values in any of the parameters, or if you specify more than 256 data shards.
The you send and receive data is a simple slice of byte slices; `[][]byte`. In the example above, the top slice must have a length of 13.
```Go
data := make([][]byte, 13)
```
You should then fill the 10 first slices with *equally sized* data, and create parity shards that will be populated with parity data. In this case we create the data in memory, but you could for instance also use [mmap](https://github.com/edsrzf/mmap-go) to map files.
```Go
// Create all shards, size them at 50000 each
for i := range input {
data[i] := make([]byte, 50000)
}
// Fill some data into the data shards
for i, in := range data[:10] {
for j:= range in {
in[j] = byte((i+j)&0xff)
}
}
```
To populate the parity shards, you simply call `Encode()` with your data.
```Go
err = enc.Encode(data)
```
The only cases where you should get an error is, if the data shards aren't of equal size. The last 3 shards now contain parity data. You can verify this by calling `Verify()`:
```Go
ok, err = enc.Verify(data)
```
The final (and important) part is to be able to reconstruct missing shards. For this to work, you need to know which parts of your data is missing. The encoder *does not know which parts are invalid*, so if data corruption is a likely scenario, you need to implement a hash check for each shard. If a byte has changed in your set, and you don't know which it is, there is no way to reconstruct the data set.
To indicate missing data, you set the shard to nil before calling `Reconstruct()`:
```Go
// Delete two data shards
data[3] = nil
data[7] = nil
// Reconstruct the missing shards
err := enc.Reconstruct(data)
```
The missing data and parity shards will be recreated. If more than 3 shards are missing, the reconstruction will fail.
So to sum up reconstruction:
* The number of data/parity shards must match the numbers used for encoding.
* The order of shards must be the same as used when encoding.
* You may only supply data you know is valid.
* Invalid shards should be set to nil.
For complete examples of an encoder and decoder see the [examples folder](https://github.com/klauspost/reedsolomon/tree/master/examples).
# Splitting/Joining Data
You might have a large slice of data. To help you split this, there are some helper functions that can split and join a single byte slice.
```Go
bigfile, _ := ioutil.Readfile("myfile.data")
// Split the file
split, err := enc.Split(bigfile)
```
This will split the file into the number of data shards set when creating the encoder and create empty parity shards.
An important thing to note is that you have to *keep track of the exact input size*. If the size of the input isn't divisible by the number of data shards, extra zeros will be inserted in the last shard.
To join a data set, use the `Join()` function, which will join the shards and write it to the `io.Writer` you supply:
```Go
// Join a data set and write it to io.Discard.
err = enc.Join(io.Discard, data, len(bigfile))
```
# Streaming/Merging
It might seem like a limitation that all data should be in memory, but an important property is that *as long as the number of data/parity shards are the same, you can merge/split data sets*, and they will remain valid as a separate set.
```Go
// Split the data set of 50000 elements into two of 25000
splitA := make([][]byte, 13)
splitB := make([][]byte, 13)
// Merge into a 100000 element set
merged := make([][]byte, 13)
for i := range data {
splitA[i] = data[i][:25000]
splitB[i] = data[i][25000:]
// Concencate it to itself
merged[i] = append(make([]byte, 0, len(data[i])*2), data[i]...)
merged[i] = append(merged[i], data[i]...)
}
// Each part should still verify as ok.
ok, err := enc.Verify(splitA)
if ok && err == nil {
log.Println("splitA ok")
}
ok, err = enc.Verify(splitB)
if ok && err == nil {
log.Println("splitB ok")
}
ok, err = enc.Verify(merge)
if ok && err == nil {
log.Println("merge ok")
}
```
This means that if you have a data set that may not fit into memory, you can split processing into smaller blocks. For the best throughput, don't use too small blocks.
This also means that you can divide big input up into smaller blocks, and do reconstruction on parts of your data. This doesn't give the same flexibility of a higher number of data shards, but it will be much more performant.
# Streaming API
There has been added support for a streaming API, to help perform fully streaming operations, which enables you to do the same operations, but on streams. To use the stream API, use [`NewStream`](https://godoc.org/github.com/klauspost/reedsolomon#NewStream) function to create the encoding/decoding interfaces. You can use [`NewStreamC`](https://godoc.org/github.com/klauspost/reedsolomon#NewStreamC) to ready an interface that reads/writes concurrently from the streams.
Input is delivered as `[]io.Reader`, output as `[]io.Writer`, and functionality corresponds to the in-memory API. Each stream must supply the same amount of data, similar to how each slice must be similar size with the in-memory API.
If an error occurs in relation to a stream, a [`StreamReadError`](https://godoc.org/github.com/klauspost/reedsolomon#StreamReadError) or [`StreamWriteError`](https://godoc.org/github.com/klauspost/reedsolomon#StreamWriteError) will help you determine which stream was the offender.
There is no buffering or timeouts/retry specified. If you want to add that, you need to add it to the Reader/Writer.
For complete examples of a streaming encoder and decoder see the [examples folder](https://github.com/klauspost/reedsolomon/tree/master/examples).
#Advanced Options
You can modify internal options which affects how jobs are split between and processed by goroutines.
To create options, use the WithXXX functions. You can supply options to `New`, `NewStream` and `NewStreamC`. If no Options are supplied, default options are used.
Example of how to supply options:
```Go
enc, err := reedsolomon.New(10, 3, WithMaxGoroutines(25))
```
# Performance
Performance depends mainly on the number of parity shards. In rough terms, doubling the number of parity shards will double the encoding time.
Here are the throughput numbers with some different selections of data and parity shards. For reference each shard is 1MB random data, and 2 CPU cores are used for encoding.
| Data | Parity | Parity | MB/s | SSSE3 MB/s | SSSE3 Speed | Rel. Speed |
|------|--------|--------|--------|-------------|-------------|------------|
| 5 | 2 | 40% | 576,11 | 2599,2 | 451% | 100,00% |
| 10 | 2 | 20% | 587,73 | 3100,28 | 528% | 102,02% |
| 10 | 4 | 40% | 298,38 | 2470,97 | 828% | 51,79% |
| 50 | 20 | 40% | 59,81 | 713,28 | 1193% | 10,38% |
If `runtime.GOMAXPROCS()` is set to a value higher than 1, the encoder will use multiple goroutines to perform the calculations in `Verify`, `Encode` and `Reconstruct`.
Example of performance scaling on Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz - 4 physical cores, 8 logical cores. The example uses 10 blocks with 16MB data each and 4 parity blocks.
| Threads | MB/s | Speed |
|---------|---------|-------|
| 1 | 1355,11 | 100% |
| 2 | 2339,78 | 172% |
| 4 | 3179,33 | 235% |
| 8 | 4346,18 | 321% |
# asm2plan9s
[asm2plan9s](https://github.com/fwessels/asm2plan9s) is used for assembling the AVX2 instructions into their BYTE/WORD/LONG equivalents.
# Links
* [Backblaze Open Sources Reed-Solomon Erasure Coding Source Code](https://www.backblaze.com/blog/reed-solomon/).
* [JavaReedSolomon](https://github.com/Backblaze/JavaReedSolomon). Compatible java library by Backblaze.
* [reedsolomon-c](https://github.com/jannson/reedsolomon-c). C version, compatible with output from this package.
* [Reed-Solomon Erasure Coding in Haskell](https://github.com/NicolasT/reedsolomon). Haskell port of the package with similar performance.
* [go-erasure](https://github.com/somethingnew2-0/go-erasure). A similar library using cgo, slower in my tests.
* [rsraid](https://github.com/goayame/rsraid). A similar library written in Go. Slower, but supports more shards.
* [Screaming Fast Galois Field Arithmetic](http://www.snia.org/sites/default/files2/SDC2013/presentations/NewThinking/EthanMiller_Screaming_Fast_Galois_Field%20Arithmetic_SIMD%20Instructions.pdf). Basis for SSE3 optimizations.
# License
This code, as the original [JavaReedSolomon](https://github.com/Backblaze/JavaReedSolomon) is published under an MIT license. See LICENSE file for more information.

20
vendor/github.com/klauspost/reedsolomon/appveyor.yml generated vendored
View File

@@ -1,20 +0,0 @@
os: Visual Studio 2015
platform: x64
clone_folder: c:\gopath\src\github.com\klauspost\reedsolomon
# environment variables
environment:
GOPATH: c:\gopath
install:
- echo %PATH%
- echo %GOPATH%
- go version
- go env
- go get -d ./...
build_script:
- go test -v -cpu=2 ./...
- go test -cpu=1,2,4 -short -race ./...

45
vendor/github.com/klauspost/reedsolomon/examples/README.md generated vendored
View File

@@ -1,45 +0,0 @@
# Examples
This folder contains usage examples of the Reed-Solomon encoder.
# Simple Encoder/Decoder
Shows basic use of the encoder, and will encode a single file into a number of
data and parity shards. This is meant as an example and is not meant for production use
since there is a number of shotcomings noted below.
To build an executable use:
```bash
go build simple-decoder.go
go build simple-encoder.go
```
# Streamin API examples
There are streaming examples of the same functionality, which streams data instead of keeping it in memory.
To build the executables use:
```bash
go build stream-decoder.go
go build stream-encoder.go
```
## Shortcomings
* If the file size of the input isn't diviable by the number of data shards
the output will contain extra zeroes
* If the shard numbers isn't the same for the decoder as in the
encoder, invalid output will be generated.
* If values have changed in a shard, it cannot be reconstructed.
* If two shards have been swapped, reconstruction will always fail.
You need to supply the shards in the same order as they were given to you.
The solution for this is to save a metadata file containing:
* File size.
* The number of data/parity shards.
* HASH of each shard.
* Order of the shards.
If you save these properties, you should abe able to detect file corruption in a shard and be able to reconstruct your data if you have the needed number of shards left.

125
vendor/github.com/klauspost/reedsolomon/examples/simple-decoder.go generated vendored
View File

@@ -1,125 +0,0 @@
//+build ignore
// Copyright 2015, Klaus Post, see LICENSE for details.
//
// Simple decoder example.
//
// The decoder reverses the process of "simple-encoder.go"
//
// To build an executable use:
//
// go build simple-decoder.go
//
// Simple Encoder/Decoder Shortcomings:
// * If the file size of the input isn't diviable by the number of data shards
// the output will contain extra zeroes
//
// * If the shard numbers isn't the same for the decoder as in the
// encoder, invalid output will be generated.
//
// * If values have changed in a shard, it cannot be reconstructed.
//
// * If two shards have been swapped, reconstruction will always fail.
// You need to supply the shards in the same order as they were given to you.
//
// The solution for this is to save a metadata file containing:
//
// * File size.
// * The number of data/parity shards.
// * HASH of each shard.
// * Order of the shards.
//
// If you save these properties, you should abe able to detect file corruption
// in a shard and be able to reconstruct your data if you have the needed number of shards left.
package main
import (
"flag"
"fmt"
"io/ioutil"
"os"
"github.com/klauspost/reedsolomon"
)
var dataShards = flag.Int("data", 4, "Number of shards to split the data into")
var parShards = flag.Int("par", 2, "Number of parity shards")
var outFile = flag.String("out", "", "Alternative output path/file")
func init() {
flag.Usage = func() {
fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0])
fmt.Fprintf(os.Stderr, " simple-decoder [-flags] basefile.ext\nDo not add the number to the filename.\n")
fmt.Fprintf(os.Stderr, "Valid flags:\n")
flag.PrintDefaults()
}
}
func main() {
// Parse flags
flag.Parse()
args := flag.Args()
if len(args) != 1 {
fmt.Fprintf(os.Stderr, "Error: No filenames given\n")
flag.Usage()
os.Exit(1)
}
fname := args[0]
// Create matrix
enc, err := reedsolomon.New(*dataShards, *parShards)
checkErr(err)
// Create shards and load the data.
shards := make([][]byte, *dataShards+*parShards)
for i := range shards {
infn := fmt.Sprintf("%s.%d", fname, i)
fmt.Println("Opening", infn)
shards[i], err = ioutil.ReadFile(infn)
if err != nil {
fmt.Println("Error reading file", err)
shards[i] = nil
}
}
// Verify the shards
ok, err := enc.Verify(shards)
if ok {
fmt.Println("No reconstruction needed")
} else {
fmt.Println("Verification failed. Reconstructing data")
err = enc.Reconstruct(shards)
if err != nil {
fmt.Println("Reconstruct failed -", err)
os.Exit(1)
}
ok, err = enc.Verify(shards)
if !ok {
fmt.Println("Verification failed after reconstruction, data likely corrupted.")
os.Exit(1)
}
checkErr(err)
}
// Join the shards and write them
outfn := *outFile
if outfn == "" {
outfn = fname
}
fmt.Println("Writing data to", outfn)
f, err := os.Create(outfn)
checkErr(err)
// We don't know the exact filesize.
err = enc.Join(f, shards, len(shards[0])**dataShards)
checkErr(err)
}
func checkErr(err error) {
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %s", err.Error())
os.Exit(2)
}
}

112
vendor/github.com/klauspost/reedsolomon/examples/simple-encoder.go generated vendored
View File

@@ -1,112 +0,0 @@
//+build ignore
// Copyright 2015, Klaus Post, see LICENSE for details.
//
// Simple encoder example
//
// The encoder encodes a simgle file into a number of shards
// To reverse the process see "simpledecoder.go"
//
// To build an executable use:
//
// go build simple-decoder.go
//
// Simple Encoder/Decoder Shortcomings:
// * If the file size of the input isn't diviable by the number of data shards
// the output will contain extra zeroes
//
// * If the shard numbers isn't the same for the decoder as in the
// encoder, invalid output will be generated.
//
// * If values have changed in a shard, it cannot be reconstructed.
//
// * If two shards have been swapped, reconstruction will always fail.
// You need to supply the shards in the same order as they were given to you.
//
// The solution for this is to save a metadata file containing:
//
// * File size.
// * The number of data/parity shards.
// * HASH of each shard.
// * Order of the shards.
//
// If you save these properties, you should abe able to detect file corruption
// in a shard and be able to reconstruct your data if you have the needed number of shards left.
package main
import (
"flag"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"github.com/klauspost/reedsolomon"
)
var dataShards = flag.Int("data", 4, "Number of shards to split the data into, must be below 257.")
var parShards = flag.Int("par", 2, "Number of parity shards")
var outDir = flag.String("out", "", "Alternative output directory")
func init() {
flag.Usage = func() {
fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0])
fmt.Fprintf(os.Stderr, " simple-encoder [-flags] filename.ext\n\n")
fmt.Fprintf(os.Stderr, "Valid flags:\n")
flag.PrintDefaults()
}
}
func main() {
// Parse command line parameters.
flag.Parse()
args := flag.Args()
if len(args) != 1 {
fmt.Fprintf(os.Stderr, "Error: No input filename given\n")
flag.Usage()
os.Exit(1)
}
if *dataShards > 257 {
fmt.Fprintf(os.Stderr, "Error: Too many data shards\n")
os.Exit(1)
}
fname := args[0]
// Create encoding matrix.
enc, err := reedsolomon.New(*dataShards, *parShards)
checkErr(err)
fmt.Println("Opening", fname)
b, err := ioutil.ReadFile(fname)
checkErr(err)
// Split the file into equally sized shards.
shards, err := enc.Split(b)
checkErr(err)
fmt.Printf("File split into %d data+parity shards with %d bytes/shard.\n", len(shards), len(shards[0]))
// Encode parity
err = enc.Encode(shards)
checkErr(err)
// Write out the resulting files.
dir, file := filepath.Split(fname)
if *outDir != "" {
dir = *outDir
}
for i, shard := range shards {
outfn := fmt.Sprintf("%s.%d", file, i)
fmt.Println("Writing to", outfn)
err = ioutil.WriteFile(filepath.Join(dir, outfn), shard, os.ModePerm)
checkErr(err)
}
}
func checkErr(err error) {
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %s", err.Error())
os.Exit(2)
}
}

167
vendor/github.com/klauspost/reedsolomon/examples/stream-decoder.go generated vendored
View File

@@ -1,167 +0,0 @@
//+build ignore
// Copyright 2015, Klaus Post, see LICENSE for details.
//
// Stream decoder example.
//
// The decoder reverses the process of "stream-encoder.go"
//
// To build an executable use:
//
// go build stream-decoder.go
//
// Simple Encoder/Decoder Shortcomings:
// * If the file size of the input isn't dividable by the number of data shards
// the output will contain extra zeroes
//
// * If the shard numbers isn't the same for the decoder as in the
// encoder, invalid output will be generated.
//
// * If values have changed in a shard, it cannot be reconstructed.
//
// * If two shards have been swapped, reconstruction will always fail.
// You need to supply the shards in the same order as they were given to you.
//
// The solution for this is to save a metadata file containing:
//
// * File size.
// * The number of data/parity shards.
// * HASH of each shard.
// * Order of the shards.
//
// If you save these properties, you should abe able to detect file corruption
// in a shard and be able to reconstruct your data if you have the needed number of shards left.
package main
import (
"flag"
"fmt"
"io"
"os"
"path/filepath"
"github.com/klauspost/reedsolomon"
)
var dataShards = flag.Int("data", 4, "Number of shards to split the data into")
var parShards = flag.Int("par", 2, "Number of parity shards")
var outFile = flag.String("out", "", "Alternative output path/file")
func init() {
flag.Usage = func() {
fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0])
fmt.Fprintf(os.Stderr, " %s [-flags] basefile.ext\nDo not add the number to the filename.\n", os.Args[0])
fmt.Fprintf(os.Stderr, "Valid flags:\n")
flag.PrintDefaults()
}
}
func main() {
// Parse flags
flag.Parse()
args := flag.Args()
if len(args) != 1 {
fmt.Fprintf(os.Stderr, "Error: No filenames given\n")
flag.Usage()
os.Exit(1)
}
fname := args[0]
// Create matrix
enc, err := reedsolomon.NewStream(*dataShards, *parShards)
checkErr(err)
// Open the inputs
shards, size, err := openInput(*dataShards, *parShards, fname)
checkErr(err)
// Verify the shards
ok, err := enc.Verify(shards)
if ok {
fmt.Println("No reconstruction needed")
} else {
fmt.Println("Verification failed. Reconstructing data")
shards, size, err = openInput(*dataShards, *parShards, fname)
checkErr(err)
// Create out destination writers
out := make([]io.Writer, len(shards))
for i := range out {
if shards[i] == nil {
dir, _ := filepath.Split(fname)
outfn := fmt.Sprintf("%s.%d", fname, i)
fmt.Println("Creating", outfn)
out[i], err = os.Create(filepath.Join(dir, outfn))
checkErr(err)
}
}
err = enc.Reconstruct(shards, out)
if err != nil {
fmt.Println("Reconstruct failed -", err)
os.Exit(1)
}
// Close output.
for i := range out {
if out[i] != nil {
err := out[i].(*os.File).Close()
checkErr(err)
}
}
shards, size, err = openInput(*dataShards, *parShards, fname)
ok, err = enc.Verify(shards)
if !ok {
fmt.Println("Verification failed after reconstruction, data likely corrupted:", err)
os.Exit(1)
}
checkErr(err)
}
// Join the shards and write them
outfn := *outFile
if outfn == "" {
outfn = fname
}
fmt.Println("Writing data to", outfn)
f, err := os.Create(outfn)
checkErr(err)
shards, size, err = openInput(*dataShards, *parShards, fname)
checkErr(err)
// We don't know the exact filesize.
err = enc.Join(f, shards, int64(*dataShards)*size)
checkErr(err)
}
func openInput(dataShards, parShards int, fname string) (r []io.Reader, size int64, err error) {
// Create shards and load the data.
shards := make([]io.Reader, dataShards+parShards)
for i := range shards {
infn := fmt.Sprintf("%s.%d", fname, i)
fmt.Println("Opening", infn)
f, err := os.Open(infn)
if err != nil {
fmt.Println("Error reading file", err)
shards[i] = nil
continue
} else {
shards[i] = f
}
stat, err := f.Stat()
checkErr(err)
if stat.Size() > 0 {
size = stat.Size()
} else {
shards[i] = nil
}
}
return shards, size, nil
}
func checkErr(err error) {
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %s", err.Error())
os.Exit(2)
}
}

142
vendor/github.com/klauspost/reedsolomon/examples/stream-encoder.go generated vendored
View File

@@ -1,142 +0,0 @@
//+build ignore
// Copyright 2015, Klaus Post, see LICENSE for details.
//
// Simple stream encoder example
//
// The encoder encodes a single file into a number of shards
// To reverse the process see "stream-decoder.go"
//
// To build an executable use:
//
// go build stream-encoder.go
//
// Simple Encoder/Decoder Shortcomings:
// * If the file size of the input isn't dividable by the number of data shards
// the output will contain extra zeroes
//
// * If the shard numbers isn't the same for the decoder as in the
// encoder, invalid output will be generated.
//
// * If values have changed in a shard, it cannot be reconstructed.
//
// * If two shards have been swapped, reconstruction will always fail.
// You need to supply the shards in the same order as they were given to you.
//
// The solution for this is to save a metadata file containing:
//
// * File size.
// * The number of data/parity shards.
// * HASH of each shard.
// * Order of the shards.
//
// If you save these properties, you should abe able to detect file corruption
// in a shard and be able to reconstruct your data if you have the needed number of shards left.
package main
import (
"flag"
"fmt"
"os"
"path/filepath"
"io"
"github.com/klauspost/reedsolomon"
)
var dataShards = flag.Int("data", 4, "Number of shards to split the data into, must be below 257.")
var parShards = flag.Int("par", 2, "Number of parity shards")
var outDir = flag.String("out", "", "Alternative output directory")
func init() {
flag.Usage = func() {
fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0])
fmt.Fprintf(os.Stderr, " %s [-flags] filename.ext\n\n", os.Args[0])
fmt.Fprintf(os.Stderr, "Valid flags:\n")
flag.PrintDefaults()
}
}
func main() {
// Parse command line parameters.
flag.Parse()
args := flag.Args()
if len(args) != 1 {
fmt.Fprintf(os.Stderr, "Error: No input filename given\n")
flag.Usage()
os.Exit(1)
}
if *dataShards > 257 {
fmt.Fprintf(os.Stderr, "Error: Too many data shards\n")
os.Exit(1)
}
fname := args[0]
// Create encoding matrix.
enc, err := reedsolomon.NewStream(*dataShards, *parShards)
checkErr(err)
fmt.Println("Opening", fname)
f, err := os.Open(fname)
checkErr(err)
instat, err := f.Stat()
checkErr(err)
shards := *dataShards + *parShards
out := make([]*os.File, shards)
// Create the resulting files.
dir, file := filepath.Split(fname)
if *outDir != "" {
dir = *outDir
}
for i := range out {
outfn := fmt.Sprintf("%s.%d", file, i)
fmt.Println("Creating", outfn)
out[i], err = os.Create(filepath.Join(dir, outfn))
checkErr(err)
}
// Split into files.
data := make([]io.Writer, *dataShards)
for i := range data {
data[i] = out[i]
}
// Do the split
err = enc.Split(f, data, instat.Size())
checkErr(err)
// Close and re-open the files.
input := make([]io.Reader, *dataShards)
for i := range data {
out[i].Close()
f, err := os.Open(out[i].Name())
checkErr(err)
input[i] = f
defer f.Close()
}
// Create parity output writers
parity := make([]io.Writer, *parShards)
for i := range parity {
parity[i] = out[*dataShards+i]
defer out[*dataShards+i].Close()
}
// Encode parity
err = enc.Encode(input, parity)
checkErr(err)
fmt.Printf("File split into %d data + %d parity shards.\n", *dataShards, *parShards)
}
func checkErr(err error) {
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %s", err.Error())
os.Exit(2)
}
}

209
vendor/github.com/klauspost/reedsolomon/examples_test.go generated vendored
View File

@@ -1,209 +0,0 @@
package reedsolomon_test
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"log"
"math/rand"
"github.com/klauspost/reedsolomon"
)
func fillRandom(p []byte) {
for i := 0; i < len(p); i += 7 {
val := rand.Int63()
for j := 0; i+j < len(p) && j < 7; j++ {
p[i+j] = byte(val)
val >>= 8
}
}
}
// Simple example of how to use all functions of the Encoder.
// Note that all error checks have been removed to keep it short.
func ExampleEncoder() {
// Create some sample data
var data = make([]byte, 250000)
fillRandom(data)
// Create an encoder with 17 data and 3 parity slices.
enc, _ := reedsolomon.New(17, 3)
// Split the data into shards
shards, _ := enc.Split(data)
// Encode the parity set
_ = enc.Encode(shards)
// Verify the parity set
ok, _ := enc.Verify(shards)
if ok {
fmt.Println("ok")
}
// Delete two shards
shards[10], shards[11] = nil, nil
// Reconstruct the shards
_ = enc.Reconstruct(shards)
// Verify the data set
ok, _ = enc.Verify(shards)
if ok {
fmt.Println("ok")
}
// Output: ok
// ok
}
// This demonstrates that shards can be arbitrary sliced and
// merged and still remain valid.
func ExampleEncoder_slicing() {
// Create some sample data
var data = make([]byte, 250000)
fillRandom(data)
// Create 5 data slices of 50000 elements each
enc, _ := reedsolomon.New(5, 3)
shards, _ := enc.Split(data)
err := enc.Encode(shards)
if err != nil {
panic(err)
}
// Check that it verifies
ok, err := enc.Verify(shards)
if ok && err == nil {
fmt.Println("encode ok")
}
// Split the data set of 50000 elements into two of 25000
splitA := make([][]byte, 8)
splitB := make([][]byte, 8)
// Merge into a 100000 element set
merged := make([][]byte, 8)
// Split/merge the shards
for i := range shards {
splitA[i] = shards[i][:25000]
splitB[i] = shards[i][25000:]
// Concencate it to itself
merged[i] = append(make([]byte, 0, len(shards[i])*2), shards[i]...)
merged[i] = append(merged[i], shards[i]...)
}
// Each part should still verify as ok.
ok, err = enc.Verify(shards)
if ok && err == nil {
fmt.Println("splitA ok")
}
ok, err = enc.Verify(splitB)
if ok && err == nil {
fmt.Println("splitB ok")
}
ok, err = enc.Verify(merged)
if ok && err == nil {
fmt.Println("merge ok")
}
// Output: encode ok
// splitA ok
// splitB ok
// merge ok
}
// This demonstrates that shards can xor'ed and
// still remain a valid set.
//
// The xor value must be the same for element 'n' in each shard,
// except if you xor with a similar sized encoded shard set.
func ExampleEncoder_xor() {
// Create some sample data
var data = make([]byte, 25000)
fillRandom(data)
// Create 5 data slices of 5000 elements each
enc, _ := reedsolomon.New(5, 3)
shards, _ := enc.Split(data)
err := enc.Encode(shards)
if err != nil {
panic(err)
}
// Check that it verifies
ok, err := enc.Verify(shards)
if !ok || err != nil {
fmt.Println("falied initial verify", err)
}
// Create an xor'ed set
xored := make([][]byte, 8)
// We xor by the index, so you can see that the xor can change,
// It should however be constant vertically through your slices.
for i := range shards {
xored[i] = make([]byte, len(shards[i]))
for j := range xored[i] {
xored[i][j] = shards[i][j] ^ byte(j&0xff)
}
}
// Each part should still verify as ok.
ok, err = enc.Verify(xored)
if ok && err == nil {
fmt.Println("verified ok after xor")
}
// Output: verified ok after xor
}
// This will show a simple stream encoder where we encode from
// a []io.Reader which contain a reader for each shard.
//
// Input and output can be exchanged with files, network streams
// or what may suit your needs.
func ExampleStreamEncoder() {
dataShards := 5
parityShards := 2
// Create a StreamEncoder with the number of data and
// parity shards.
rs, err := reedsolomon.NewStream(dataShards, parityShards)
if err != nil {
log.Fatal(err)
}
shardSize := 50000
// Create input data shards.
input := make([][]byte, dataShards)
for s := range input {
input[s] = make([]byte, shardSize)
fillRandom(input[s])
}
// Convert our buffers to io.Readers
readers := make([]io.Reader, dataShards)
for i := range readers {
readers[i] = io.Reader(bytes.NewBuffer(input[i]))
}
// Create our output io.Writers
out := make([]io.Writer, parityShards)
for i := range out {
out[i] = ioutil.Discard
}
// Encode from input to output.
err = rs.Encode(readers, out)
if err != nil {
log.Fatal(err)
}
fmt.Println("ok")
// OUTPUT: ok
}

134
vendor/github.com/klauspost/reedsolomon/galois.go generated vendored
View File

File diff suppressed because one or more lines are too long

73
vendor/github.com/klauspost/reedsolomon/galois_amd64.go generated vendored
View File

@@ -1,73 +0,0 @@
//+build !noasm
//+build !appengine
// Copyright 2015, Klaus Post, see LICENSE for details.
package reedsolomon
//go:noescape
func galMulSSSE3(low, high, in, out []byte)
//go:noescape
func galMulSSSE3Xor(low, high, in, out []byte)
//go:noescape
func galMulAVX2Xor(low, high, in, out []byte)
//go:noescape
func galMulAVX2(low, high, in, out []byte)
// This is what the assembler rountes does in blocks of 16 bytes:
/*
func galMulSSSE3(low, high, in, out []byte) {
for n, input := range in {
l := input & 0xf
h := input >> 4
out[n] = low[l] ^ high[h]
}
}
func galMulSSSE3Xor(low, high, in, out []byte) {
for n, input := range in {
l := input & 0xf
h := input >> 4
out[n] ^= low[l] ^ high[h]
}
}
*/
func galMulSlice(c byte, in, out []byte, ssse3, avx2 bool) {
var done int
if avx2 {
galMulAVX2(mulTableLow[c][:], mulTableHigh[c][:], in, out)
done = (len(in) >> 5) << 5
} else if ssse3 {
galMulSSSE3(mulTableLow[c][:], mulTableHigh[c][:], in, out)
done = (len(in) >> 4) << 4
}
remain := len(in) - done
if remain > 0 {
mt := mulTable[c]
for i := done; i < len(in); i++ {
out[i] = mt[in[i]]
}
}
}
func galMulSliceXor(c byte, in, out []byte, ssse3, avx2 bool) {
var done int
if avx2 {
galMulAVX2Xor(mulTableLow[c][:], mulTableHigh[c][:], in, out)
done = (len(in) >> 5) << 5
} else if ssse3 {
galMulSSSE3Xor(mulTableLow[c][:], mulTableHigh[c][:], in, out)
done = (len(in) >> 4) << 4
}
remain := len(in) - done
if remain > 0 {
mt := mulTable[c]
for i := done; i < len(in); i++ {
out[i] ^= mt[in[i]]
}
}
}

164
vendor/github.com/klauspost/reedsolomon/galois_amd64.s generated vendored
View File

@@ -1,164 +0,0 @@
//+build !noasm !appengine
// Copyright 2015, Klaus Post, see LICENSE for details.
// Based on http://www.snia.org/sites/default/files2/SDC2013/presentations/NewThinking/EthanMiller_Screaming_Fast_Galois_Field%20Arithmetic_SIMD%20Instructions.pdf
// and http://jerasure.org/jerasure/gf-complete/tree/master
// func galMulSSSE3Xor(low, high, in, out []byte)
TEXT ·galMulSSSE3Xor(SB), 7, $0
MOVQ low+0(FP), SI // SI: &low
MOVQ high+24(FP), DX // DX: &high
MOVOU (SI), X6 // X6 low
MOVOU (DX), X7 // X7: high
MOVQ $15, BX // BX: low mask
MOVQ BX, X8
PXOR X5, X5
MOVQ in+48(FP), SI // R11: &in
MOVQ in_len+56(FP), R9 // R9: len(in)
MOVQ out+72(FP), DX // DX: &out
PSHUFB X5, X8 // X8: lomask (unpacked)
SHRQ $4, R9 // len(in) / 16
CMPQ R9, $0
JEQ done_xor
loopback_xor:
MOVOU (SI), X0 // in[x]
MOVOU (DX), X4 // out[x]
MOVOU X0, X1 // in[x]
MOVOU X6, X2 // low copy
MOVOU X7, X3 // high copy
PSRLQ $4, X1 // X1: high input
PAND X8, X0 // X0: low input
PAND X8, X1 // X0: high input
PSHUFB X0, X2 // X2: mul low part
PSHUFB X1, X3 // X3: mul high part
PXOR X2, X3 // X3: Result
PXOR X4, X3 // X3: Result xor existing out
MOVOU X3, (DX) // Store
ADDQ $16, SI // in+=16
ADDQ $16, DX // out+=16
SUBQ $1, R9
JNZ loopback_xor
done_xor:
RET
// func galMulSSSE3(low, high, in, out []byte)
TEXT ·galMulSSSE3(SB), 7, $0
MOVQ low+0(FP), SI // SI: &low
MOVQ high+24(FP), DX // DX: &high
MOVOU (SI), X6 // X6 low
MOVOU (DX), X7 // X7: high
MOVQ $15, BX // BX: low mask
MOVQ BX, X8
PXOR X5, X5
MOVQ in+48(FP), SI // R11: &in
MOVQ in_len+56(FP), R9 // R9: len(in)
MOVQ out+72(FP), DX // DX: &out
PSHUFB X5, X8 // X8: lomask (unpacked)
SHRQ $4, R9 // len(in) / 16
CMPQ R9, $0
JEQ done
loopback:
MOVOU (SI), X0 // in[x]
MOVOU X0, X1 // in[x]
MOVOU X6, X2 // low copy
MOVOU X7, X3 // high copy
PSRLQ $4, X1 // X1: high input
PAND X8, X0 // X0: low input
PAND X8, X1 // X0: high input
PSHUFB X0, X2 // X2: mul low part
PSHUFB X1, X3 // X3: mul high part
PXOR X2, X3 // X3: Result
MOVOU X3, (DX) // Store
ADDQ $16, SI // in+=16
ADDQ $16, DX // out+=16
SUBQ $1, R9
JNZ loopback
done:
RET
// func galMulAVX2Xor(low, high, in, out []byte)
TEXT ·galMulAVX2Xor(SB), 7, $0
MOVQ low+0(FP), SI // SI: &low
MOVQ high+24(FP), DX // DX: &high
MOVQ $15, BX // BX: low mask
MOVQ BX, X5
MOVOU (SI), X6 // X6 low
MOVOU (DX), X7 // X7: high
MOVQ in_len+56(FP), R9 // R9: len(in)
LONG $0x384de3c4; WORD $0x01f6 // VINSERTI128 YMM6, YMM6, XMM6, 1 ; low
LONG $0x3845e3c4; WORD $0x01ff // VINSERTI128 YMM7, YMM7, XMM7, 1 ; high
LONG $0x787d62c4; BYTE $0xc5 // VPBROADCASTB YMM8, XMM5 ; X8: lomask (unpacked)
SHRQ $5, R9 // len(in) /32
MOVQ out+72(FP), DX // DX: &out
MOVQ in+48(FP), SI // R11: &in
TESTQ R9, R9
JZ done_xor_avx2
loopback_xor_avx2:
LONG $0x066ffec5 // VMOVDQU YMM0, [rsi]
LONG $0x226ffec5 // VMOVDQU YMM4, [rdx]
LONG $0xd073f5c5; BYTE $0x04 // VPSRLQ YMM1, YMM0, 4 ; X1: high input
LONG $0xdb7dc1c4; BYTE $0xc0 // VPAND YMM0, YMM0, YMM8 ; X0: low input
LONG $0xdb75c1c4; BYTE $0xc8 // VPAND YMM1, YMM1, YMM8 ; X1: high input
LONG $0x004de2c4; BYTE $0xd0 // VPSHUFB YMM2, YMM6, YMM0 ; X2: mul low part
LONG $0x0045e2c4; BYTE $0xd9 // VPSHUFB YMM3, YMM7, YMM1 ; X2: mul high part
LONG $0xdbefedc5 // VPXOR YMM3, YMM2, YMM3 ; X3: Result
LONG $0xe4efe5c5 // VPXOR YMM4, YMM3, YMM4 ; X4: Result
LONG $0x227ffec5 // VMOVDQU [rdx], YMM4
ADDQ $32, SI // in+=32
ADDQ $32, DX // out+=32
SUBQ $1, R9
JNZ loopback_xor_avx2
done_xor_avx2:
// VZEROUPPER
BYTE $0xc5; BYTE $0xf8; BYTE $0x77
RET
// func galMulAVX2(low, high, in, out []byte)
TEXT ·galMulAVX2(SB), 7, $0
MOVQ low+0(FP), SI // SI: &low
MOVQ high+24(FP), DX // DX: &high
MOVQ $15, BX // BX: low mask
MOVQ BX, X5
MOVOU (SI), X6 // X6 low
MOVOU (DX), X7 // X7: high
MOVQ in_len+56(FP), R9 // R9: len(in)
LONG $0x384de3c4; WORD $0x01f6 // VINSERTI128 YMM6, YMM6, XMM6, 1 ; low
LONG $0x3845e3c4; WORD $0x01ff // VINSERTI128 YMM7, YMM7, XMM7, 1 ; high
LONG $0x787d62c4; BYTE $0xc5 // VPBROADCASTB YMM8, XMM5 ; X8: lomask (unpacked)
SHRQ $5, R9 // len(in) /32
MOVQ out+72(FP), DX // DX: &out
MOVQ in+48(FP), SI // R11: &in
TESTQ R9, R9
JZ done_avx2
loopback_avx2:
LONG $0x066ffec5 // VMOVDQU YMM0, [rsi]
LONG $0xd073f5c5; BYTE $0x04 // VPSRLQ YMM1, YMM0, 4 ; X1: high input
LONG $0xdb7dc1c4; BYTE $0xc0 // VPAND YMM0, YMM0, YMM8 ; X0: low input
LONG $0xdb75c1c4; BYTE $0xc8 // VPAND YMM1, YMM1, YMM8 ; X1: high input
LONG $0x004de2c4; BYTE $0xd0 // VPSHUFB YMM2, YMM6, YMM0 ; X2: mul low part
LONG $0x0045e2c4; BYTE $0xd9 // VPSHUFB YMM3, YMM7, YMM1 ; X2: mul high part
LONG $0xe3efedc5 // VPXOR YMM4, YMM2, YMM3 ; X4: Result
LONG $0x227ffec5 // VMOVDQU [rdx], YMM4
ADDQ $32, SI // in+=32
ADDQ $32, DX // out+=32
SUBQ $1, R9
JNZ loopback_avx2
done_avx2:
BYTE $0xc5; BYTE $0xf8; BYTE $0x77 // VZEROUPPER
RET

19
vendor/github.com/klauspost/reedsolomon/galois_noasm.go generated vendored
View File

@@ -1,19 +0,0 @@
//+build !amd64 noasm appengine
// Copyright 2015, Klaus Post, see LICENSE for details.
package reedsolomon
func galMulSlice(c byte, in, out []byte, ssse3, avx2 bool) {
mt := mulTable[c]
for n, input := range in {
out[n] = mt[input]
}
}
func galMulSliceXor(c byte, in, out []byte, ssse3, avx2 bool) {
mt := mulTable[c]
for n, input := range in {
out[n] ^= mt[input]
}
}

155
vendor/github.com/klauspost/reedsolomon/galois_test.go generated vendored
View File

@@ -1,155 +0,0 @@
/**
* Unit tests for Galois
*
* Copyright 2015, Klaus Post
* Copyright 2015, Backblaze, Inc.
*/
package reedsolomon
import (
"bytes"
"testing"
)
func TestAssociativity(t *testing.T) {
for i := 0; i < 256; i++ {
a := byte(i)
for j := 0; j < 256; j++ {
b := byte(j)
for k := 0; k < 256; k++ {
c := byte(k)
x := galAdd(a, galAdd(b, c))
y := galAdd(galAdd(a, b), c)
if x != y {
t.Fatal("add does not match:", x, "!=", y)
}
x = galMultiply(a, galMultiply(b, c))
y = galMultiply(galMultiply(a, b), c)
if x != y {
t.Fatal("multiply does not match:", x, "!=", y)
}
}
}
}
}
func TestIdentity(t *testing.T) {
for i := 0; i < 256; i++ {
a := byte(i)
b := galAdd(a, 0)
if a != b {
t.Fatal("Add zero should yield same result", a, "!=", b)
}
b = galMultiply(a, 1)
if a != b {
t.Fatal("Mul by one should yield same result", a, "!=", b)
}
}
}
func TestInverse(t *testing.T) {
for i := 0; i < 256; i++ {
a := byte(i)
b := galSub(0, a)
c := galAdd(a, b)
if c != 0 {
t.Fatal("inverse sub/add", c, "!=", 0)
}
if a != 0 {
b = galDivide(1, a)
c = galMultiply(a, b)
if c != 1 {
t.Fatal("inverse div/mul", c, "!=", 1)
}
}
}
}
func TestCommutativity(t *testing.T) {
for i := 0; i < 256; i++ {
a := byte(i)
for j := 0; j < 256; j++ {
b := byte(j)
x := galAdd(a, b)
y := galAdd(b, a)
if x != y {
t.Fatal(x, "!= ", y)
}
x = galMultiply(a, b)
y = galMultiply(b, a)
if x != y {
t.Fatal(x, "!= ", y)
}
}
}
}
func TestDistributivity(t *testing.T) {
for i := 0; i < 256; i++ {
a := byte(i)
for j := 0; j < 256; j++ {
b := byte(j)
for k := 0; k < 256; k++ {
c := byte(k)
x := galMultiply(a, galAdd(b, c))
y := galAdd(galMultiply(a, b), galMultiply(a, c))
if x != y {
t.Fatal(x, "!= ", y)
}
}
}
}
}
func TestExp(t *testing.T) {
for i := 0; i < 256; i++ {
a := byte(i)
power := byte(1)
for j := 0; j < 256; j++ {
x := galExp(a, j)
if x != power {
t.Fatal(x, "!=", power)
}
power = galMultiply(power, a)
}
}
}
func TestGalois(t *testing.T) {
// These values were copied output of the Python code.
if galMultiply(3, 4) != 12 {
t.Fatal("galMultiply(3, 4) != 12")
}
if galMultiply(7, 7) != 21 {
t.Fatal("galMultiply(7, 7) != 21")
}
if galMultiply(23, 45) != 41 {
t.Fatal("galMultiply(23, 45) != 41")
}
// Test slices (>16 entries to test assembler)
in := []byte{0, 1, 2, 3, 4, 5, 6, 10, 50, 100, 150, 174, 201, 255, 99, 32, 67, 85}
out := make([]byte, len(in))
galMulSlice(25, in, out, false, false)
expect := []byte{0x0, 0x19, 0x32, 0x2b, 0x64, 0x7d, 0x56, 0xfa, 0xb8, 0x6d, 0xc7, 0x85, 0xc3, 0x1f, 0x22, 0x7, 0x25, 0xfe}
if 0 != bytes.Compare(out, expect) {
t.Errorf("got %#v, expected %#v", out, expect)
}
galMulSlice(177, in, out, false, false)
expect = []byte{0x0, 0xb1, 0x7f, 0xce, 0xfe, 0x4f, 0x81, 0x9e, 0x3, 0x6, 0xe8, 0x75, 0xbd, 0x40, 0x36, 0xa3, 0x95, 0xcb}
if 0 != bytes.Compare(out, expect) {
t.Errorf("got %#v, expected %#v", out, expect)
}
if galExp(2, 2) != 4 {
t.Fatal("galExp(2, 2) != 4")
}
if galExp(5, 20) != 235 {
t.Fatal("galExp(5, 20) != 235")
}
if galExp(13, 7) != 43 {
t.Fatal("galExp(13, 7) != 43")
}
}

132
vendor/github.com/klauspost/reedsolomon/gentables.go generated vendored
View File

@@ -1,132 +0,0 @@
//+build ignore
package main
import (
"fmt"
)
var logTable = [fieldSize]int16{
-1, 0, 1, 25, 2, 50, 26, 198,
3, 223, 51, 238, 27, 104, 199, 75,
4, 100, 224, 14, 52, 141, 239, 129,
28, 193, 105, 248, 200, 8, 76, 113,
5, 138, 101, 47, 225, 36, 15, 33,
53, 147, 142, 218, 240, 18, 130, 69,
29, 181, 194, 125, 106, 39, 249, 185,
201, 154, 9, 120, 77, 228, 114, 166,
6, 191, 139, 98, 102, 221, 48, 253,
226, 152, 37, 179, 16, 145, 34, 136,
54, 208, 148, 206, 143, 150, 219, 189,
241, 210, 19, 92, 131, 56, 70, 64,
30, 66, 182, 163, 195, 72, 126, 110,
107, 58, 40, 84, 250, 133, 186, 61,
202, 94, 155, 159, 10, 21, 121, 43,
78, 212, 229, 172, 115, 243, 167, 87,
7, 112, 192, 247, 140, 128, 99, 13,
103, 74, 222, 237, 49, 197, 254, 24,
227, 165, 153, 119, 38, 184, 180, 124,
17, 68, 146, 217, 35, 32, 137, 46,
55, 63, 209, 91, 149, 188, 207, 205,
144, 135, 151, 178, 220, 252, 190, 97,
242, 86, 211, 171, 20, 42, 93, 158,
132, 60, 57, 83, 71, 109, 65, 162,
31, 45, 67, 216, 183, 123, 164, 118,
196, 23, 73, 236, 127, 12, 111, 246,
108, 161, 59, 82, 41, 157, 85, 170,
251, 96, 134, 177, 187, 204, 62, 90,
203, 89, 95, 176, 156, 169, 160, 81,
11, 245, 22, 235, 122, 117, 44, 215,
79, 174, 213, 233, 230, 231, 173, 232,
116, 214, 244, 234, 168, 80, 88, 175,
}
const (
// The number of elements in the field.
fieldSize = 256
// The polynomial used to generate the logarithm table.
//
// There are a number of polynomials that work to generate
// a Galois field of 256 elements. The choice is arbitrary,
// and we just use the first one.
//
// The possibilities are: 29, 43, 45, 77, 95, 99, 101, 105,
//* 113, 135, 141, 169, 195, 207, 231, and 245.
generatingPolynomial = 29
)
func main() {
t := generateExpTable()
fmt.Printf("var expTable = %#v\n", t)
//t2 := generateMulTableSplit(t)
//fmt.Printf("var mulTable = %#v\n", t2)
low, high := generateMulTableHalf(t)
fmt.Printf("var mulTableLow = %#v\n", low)
fmt.Printf("var mulTableHigh = %#v\n", high)
}
/**
* Generates the inverse log table.
*/
func generateExpTable() []byte {
result := make([]byte, fieldSize*2-2)
for i := 1; i < fieldSize; i++ {
log := logTable[i]
result[log] = byte(i)
result[log+fieldSize-1] = byte(i)
}
return result
}
func generateMulTable(expTable []byte) []byte {
result := make([]byte, 256*256)
for v := range result {
a := byte(v & 0xff)
b := byte(v >> 8)
if a == 0 || b == 0 {
result[v] = 0
continue
}
logA := int(logTable[a])
logB := int(logTable[b])
result[v] = expTable[logA+logB]
}
return result
}
func generateMulTableSplit(expTable []byte) [256][256]byte {
var result [256][256]byte
for a := range result {
for b := range result[a] {
if a == 0 || b == 0 {
result[a][b] = 0
continue
}
logA := int(logTable[a])
logB := int(logTable[b])
result[a][b] = expTable[logA+logB]
}
}
return result
}
func generateMulTableHalf(expTable []byte) (low [256][16]byte, high [256][16]byte) {
for a := range low {
for b := range low {
result := 0
if !(a == 0 || b == 0) {
logA := int(logTable[a])
logB := int(logTable[b])
result = int(expTable[logA+logB])
}
if (b & 0xf) == b {
low[a][b] = byte(result)
}
if (b & 0xf0) == b {
high[a][b>>4] = byte(result)
}
}
}
return
}

160
vendor/github.com/klauspost/reedsolomon/inversion_tree.go generated vendored
View File

@@ -1,160 +0,0 @@
/**
* A thread-safe tree which caches inverted matrices.
*
* Copyright 2016, Peter Collins
*/
package reedsolomon
import (
"errors"
"sync"
)
// The tree uses a Reader-Writer mutex to make it thread-safe
// when accessing cached matrices and inserting new ones.
type inversionTree struct {
mutex *sync.RWMutex
root inversionNode
}
type inversionNode struct {
matrix matrix
children []*inversionNode
}
// newInversionTree initializes a tree for storing inverted matrices.
// Note that the root node is the identity matrix as it implies
// there were no errors with the original data.
func newInversionTree(dataShards, parityShards int) inversionTree {
identity, _ := identityMatrix(dataShards)
root := inversionNode{
matrix: identity,
children: make([]*inversionNode, dataShards+parityShards),
}
return inversionTree{
mutex: &sync.RWMutex{},
root: root,
}
}
// GetInvertedMatrix returns the cached inverted matrix or nil if it
// is not found in the tree keyed on the indices of invalid rows.
func (t inversionTree) GetInvertedMatrix(invalidIndices []int) matrix {
// Lock the tree for reading before accessing the tree.
t.mutex.RLock()
defer t.mutex.RUnlock()
// If no invalid indices were give we should return the root
// identity matrix.
if len(invalidIndices) == 0 {
return t.root.matrix
}
// Recursively search for the inverted matrix in the tree, passing in
// 0 as the parent index as we start at the root of the tree.
return t.root.getInvertedMatrix(invalidIndices, 0)
}
// errAlreadySet is returned if the root node matrix is overwritten
var errAlreadySet = errors.New("the root node identity matrix is already set")
// InsertInvertedMatrix inserts a new inverted matrix into the tree
// keyed by the indices of invalid rows. The total number of shards
// is required for creating the proper length lists of child nodes for
// each node.
func (t inversionTree) InsertInvertedMatrix(invalidIndices []int, matrix matrix, shards int) error {
// If no invalid indices were given then we are done because the
// root node is already set with the identity matrix.
if len(invalidIndices) == 0 {
return errAlreadySet
}
if !matrix.IsSquare() {
return errNotSquare
}
// Lock the tree for writing and reading before accessing the tree.
t.mutex.Lock()
defer t.mutex.Unlock()
// Recursively create nodes for the inverted matrix in the tree until
// we reach the node to insert the matrix to. We start by passing in
// 0 as the parent index as we start at the root of the tree.
t.root.insertInvertedMatrix(invalidIndices, matrix, shards, 0)
return nil
}
func (n inversionNode) getInvertedMatrix(invalidIndices []int, parent int) matrix {
// Get the child node to search next from the list of children. The
// list of children starts relative to the parent index passed in
// because the indices of invalid rows is sorted (by default). As we
// search recursively, the first invalid index gets popped off the list,
// so when searching through the list of children, use that first invalid
// index to find the child node.
firstIndex := invalidIndices[0]
node := n.children[firstIndex-parent]
// If the child node doesn't exist in the list yet, fail fast by
// returning, so we can construct and insert the proper inverted matrix.
if node == nil {
return nil
}
// If there's more than one invalid index left in the list we should
// keep searching recursively.
if len(invalidIndices) > 1 {
// Search recursively on the child node by passing in the invalid indices
// with the first index popped off the front. Also the parent index to
// pass down is the first index plus one.
return node.getInvertedMatrix(invalidIndices[1:], firstIndex+1)
}
// If there aren't any more invalid indices to search, we've found our
// node. Return it, however keep in mind that the matrix could still be
// nil because intermediary nodes in the tree are created sometimes with
// their inversion matrices uninitialized.
return node.matrix
}
func (n inversionNode) insertInvertedMatrix(invalidIndices []int, matrix matrix, shards, parent int) {
// As above, get the child node to search next from the list of children.
// The list of children starts relative to the parent index passed in
// because the indices of invalid rows is sorted (by default). As we
// search recursively, the first invalid index gets popped off the list,
// so when searching through the list of children, use that first invalid
// index to find the child node.
firstIndex := invalidIndices[0]
node := n.children[firstIndex-parent]
// If the child node doesn't exist in the list yet, create a new
// node because we have the writer lock and add it to the list
// of children.
if node == nil {
// Make the length of the list of children equal to the number
// of shards minus the first invalid index because the list of
// invalid indices is sorted, so only this length of errors
// are possible in the tree.
node = &inversionNode{
children: make([]*inversionNode, shards-firstIndex),
}
// Insert the new node into the tree at the first index relative
// to the parent index that was given in this recursive call.
n.children[firstIndex-parent] = node
}
// If there's more than one invalid index left in the list we should
// keep searching recursively in order to find the node to add our
// matrix.
if len(invalidIndices) > 1 {
// As above, search recursively on the child node by passing in
// the invalid indices with the first index popped off the front.
// Also the total number of shards and parent index are passed down
// which is equal to the first index plus one.
node.insertInvertedMatrix(invalidIndices[1:], matrix, shards, firstIndex+1)
} else {
// If there aren't any more invalid indices to search, we've found our
// node. Cache the inverted matrix in this node.
node.matrix = matrix
}
}

125
vendor/github.com/klauspost/reedsolomon/inversion_tree_test.go generated vendored
View File

@@ -1,125 +0,0 @@
/**
* Unit tests for inversion tree.
*
* Copyright 2016, Peter Collins
*/
package reedsolomon
import (
"testing"
)
func TestNewInversionTree(t *testing.T) {
tree := newInversionTree(3, 2)
children := len(tree.root.children)
if children != 5 {
t.Fatal("Root node children list length", children, "!=", 5)
}
str := tree.root.matrix.String()
expect := "[[1, 0, 0], [0, 1, 0], [0, 0, 1]]"
if str != expect {
t.Fatal(str, "!=", expect)
}
}
func TestGetInvertedMatrix(t *testing.T) {
tree := newInversionTree(3, 2)
matrix := tree.GetInvertedMatrix([]int{})
str := matrix.String()
expect := "[[1, 0, 0], [0, 1, 0], [0, 0, 1]]"
if str != expect {
t.Fatal(str, "!=", expect)
}
matrix = tree.GetInvertedMatrix([]int{1})
if matrix != nil {
t.Fatal(matrix, "!= nil")
}
matrix = tree.GetInvertedMatrix([]int{1, 2})
if matrix != nil {
t.Fatal(matrix, "!= nil")
}
matrix, err := newMatrix(3, 3)
if err != nil {
t.Fatalf("Failed initializing new Matrix : %s", err)
}
err = tree.InsertInvertedMatrix([]int{1}, matrix, 5)
if err != nil {
t.Fatalf("Failed inserting new Matrix : %s", err)
}
cachedMatrix := tree.GetInvertedMatrix([]int{1})
if cachedMatrix == nil {
t.Fatal(cachedMatrix, "== nil")
}
if matrix.String() != cachedMatrix.String() {
t.Fatal(matrix.String(), "!=", cachedMatrix.String())
}
}
func TestInsertInvertedMatrix(t *testing.T) {
tree := newInversionTree(3, 2)
matrix, err := newMatrix(3, 3)
if err != nil {
t.Fatalf("Failed initializing new Matrix : %s", err)
}
err = tree.InsertInvertedMatrix([]int{1}, matrix, 5)
if err != nil {
t.Fatalf("Failed inserting new Matrix : %s", err)
}
err = tree.InsertInvertedMatrix([]int{}, matrix, 5)
if err == nil {
t.Fatal("Should have failed inserting the root node matrix", matrix)
}
matrix, err = newMatrix(3, 2)
if err != nil {
t.Fatalf("Failed initializing new Matrix : %s", err)
}
err = tree.InsertInvertedMatrix([]int{2}, matrix, 5)
if err == nil {
t.Fatal("Should have failed inserting a non-square matrix", matrix)
}
matrix, err = newMatrix(3, 3)
if err != nil {
t.Fatalf("Failed initializing new Matrix : %s", err)
}
err = tree.InsertInvertedMatrix([]int{0, 1}, matrix, 5)
if err != nil {
t.Fatalf("Failed inserting new Matrix : %s", err)
}
}
func TestDoubleInsertInvertedMatrix(t *testing.T) {
tree := newInversionTree(3, 2)
matrix, err := newMatrix(3, 3)
if err != nil {
t.Fatalf("Failed initializing new Matrix : %s", err)
}
err = tree.InsertInvertedMatrix([]int{1}, matrix, 5)
if err != nil {
t.Fatalf("Failed inserting new Matrix : %s", err)
}
err = tree.InsertInvertedMatrix([]int{1}, matrix, 5)
if err != nil {
t.Fatalf("Failed inserting new Matrix : %s", err)
}
cachedMatrix := tree.GetInvertedMatrix([]int{1})
if cachedMatrix == nil {
t.Fatal(cachedMatrix, "== nil")
}
if matrix.String() != cachedMatrix.String() {
t.Fatal(matrix.String(), "!=", cachedMatrix.String())
}
}

279
vendor/github.com/klauspost/reedsolomon/matrix.go generated vendored
View File

@@ -1,279 +0,0 @@
/**
* Matrix Algebra over an 8-bit Galois Field
*
* Copyright 2015, Klaus Post
* Copyright 2015, Backblaze, Inc.
*/
package reedsolomon
import (
"errors"
"fmt"
"strconv"
"strings"
)
// byte[row][col]
type matrix [][]byte
// newMatrix returns a matrix of zeros.
func newMatrix(rows, cols int) (matrix, error) {
if rows <= 0 {
return nil, errInvalidRowSize
}
if cols <= 0 {
return nil, errInvalidColSize
}
m := matrix(make([][]byte, rows))
for i := range m {
m[i] = make([]byte, cols)
}
return m, nil
}
// NewMatrixData initializes a matrix with the given row-major data.
// Note that data is not copied from input.
func newMatrixData(data [][]byte) (matrix, error) {
m := matrix(data)
err := m.Check()
if err != nil {
return nil, err
}
return m, nil
}
// IdentityMatrix returns an identity matrix of the given size.
func identityMatrix(size int) (matrix, error) {
m, err := newMatrix(size, size)
if err != nil {
return nil, err
}
for i := range m {
m[i][i] = 1
}
return m, nil
}
// errInvalidRowSize will be returned if attempting to create a matrix with negative or zero row number.
var errInvalidRowSize = errors.New("invalid row size")
// errInvalidColSize will be returned if attempting to create a matrix with negative or zero column number.
var errInvalidColSize = errors.New("invalid column size")
// errColSizeMismatch is returned if the size of matrix columns mismatch.
var errColSizeMismatch = errors.New("column size is not the same for all rows")
func (m matrix) Check() error {
rows := len(m)
if rows <= 0 {
return errInvalidRowSize
}
cols := len(m[0])
if cols <= 0 {
return errInvalidColSize
}
for _, col := range m {
if len(col) != cols {
return errColSizeMismatch
}
}
return nil
}
// String returns a human-readable string of the matrix contents.
//
// Example: [[1, 2], [3, 4]]
func (m matrix) String() string {
rowOut := make([]string, 0, len(m))
for _, row := range m {
colOut := make([]string, 0, len(row))
for _, col := range row {
colOut = append(colOut, strconv.Itoa(int(col)))
}
rowOut = append(rowOut, "["+strings.Join(colOut, ", ")+"]")
}
return "[" + strings.Join(rowOut, ", ") + "]"
}
// Multiply multiplies this matrix (the one on the left) by another
// matrix (the one on the right) and returns a new matrix with the result.
func (m matrix) Multiply(right matrix) (matrix, error) {
if len(m[0]) != len(right) {
return nil, fmt.Errorf("columns on left (%d) is different than rows on right (%d)", len(m[0]), len(right))
}
result, _ := newMatrix(len(m), len(right[0]))
for r, row := range result {
for c := range row {
var value byte
for i := range m[0] {
value ^= galMultiply(m[r][i], right[i][c])
}
result[r][c] = value
}
}
return result, nil
}
// Augment returns the concatenation of this matrix and the matrix on the right.
func (m matrix) Augment(right matrix) (matrix, error) {
if len(m) != len(right) {
return nil, errMatrixSize
}
result, _ := newMatrix(len(m), len(m[0])+len(right[0]))
for r, row := range m {
for c := range row {
result[r][c] = m[r][c]
}
cols := len(m[0])
for c := range right[0] {
result[r][cols+c] = right[r][c]
}
}
return result, nil
}
// errMatrixSize is returned if matrix dimensions are doesn't match.
var errMatrixSize = errors.New("matrix sizes does not match")
func (m matrix) SameSize(n matrix) error {
if len(m) != len(n) {
return errMatrixSize
}
for i := range m {
if len(m[i]) != len(n[i]) {
return errMatrixSize
}
}
return nil
}
// Returns a part of this matrix. Data is copied.
func (m matrix) SubMatrix(rmin, cmin, rmax, cmax int) (matrix, error) {
result, err := newMatrix(rmax-rmin, cmax-cmin)
if err != nil {
return nil, err
}
// OPTME: If used heavily, use copy function to copy slice
for r := rmin; r < rmax; r++ {
for c := cmin; c < cmax; c++ {
result[r-rmin][c-cmin] = m[r][c]
}
}
return result, nil
}
// SwapRows Exchanges two rows in the matrix.
func (m matrix) SwapRows(r1, r2 int) error {
if r1 < 0 || len(m) <= r1 || r2 < 0 || len(m) <= r2 {
return errInvalidRowSize
}
m[r2], m[r1] = m[r1], m[r2]
return nil
}
// IsSquare will return true if the matrix is square
// and nil if the matrix is square
func (m matrix) IsSquare() bool {
return len(m) == len(m[0])
}
// errSingular is returned if the matrix is singular and cannot be inversed
var errSingular = errors.New("matrix is singular")
// errNotSquare is returned if attempting to inverse a non-square matrix.
var errNotSquare = errors.New("only square matrices can be inverted")
// Invert returns the inverse of this matrix.
// Returns ErrSingular when the matrix is singular and doesn't have an inverse.
// The matrix must be square, otherwise ErrNotSquare is returned.
func (m matrix) Invert() (matrix, error) {
if !m.IsSquare() {
return nil, errNotSquare
}
size := len(m)
work, _ := identityMatrix(size)
work, _ = m.Augment(work)
err := work.gaussianElimination()
if err != nil {
return nil, err
}
return work.SubMatrix(0, size, size, size*2)
}
func (m matrix) gaussianElimination() error {
rows := len(m)
columns := len(m[0])
// Clear out the part below the main diagonal and scale the main
// diagonal to be 1.
for r := 0; r < rows; r++ {
// If the element on the diagonal is 0, find a row below
// that has a non-zero and swap them.
if m[r][r] == 0 {
for rowBelow := r + 1; rowBelow < rows; rowBelow++ {
if m[rowBelow][r] != 0 {
m.SwapRows(r, rowBelow)
break
}
}
}
// If we couldn't find one, the matrix is singular.
if m[r][r] == 0 {
return errSingular
}
// Scale to 1.
if m[r][r] != 1 {
scale := galDivide(1, m[r][r])
for c := 0; c < columns; c++ {
m[r][c] = galMultiply(m[r][c], scale)
}
}
// Make everything below the 1 be a 0 by subtracting
// a multiple of it. (Subtraction and addition are
// both exclusive or in the Galois field.)
for rowBelow := r + 1; rowBelow < rows; rowBelow++ {
if m[rowBelow][r] != 0 {
scale := m[rowBelow][r]
for c := 0; c < columns; c++ {
m[rowBelow][c] ^= galMultiply(scale, m[r][c])
}
}
}
}
// Now clear the part above the main diagonal.
for d := 0; d < rows; d++ {
for rowAbove := 0; rowAbove < d; rowAbove++ {
if m[rowAbove][d] != 0 {
scale := m[rowAbove][d]
for c := 0; c < columns; c++ {
m[rowAbove][c] ^= galMultiply(scale, m[d][c])
}
}
}
}
return nil
}
// Create a Vandermonde matrix, which is guaranteed to have the
// property that any subset of rows that forms a square matrix
// is invertible.
func vandermonde(rows, cols int) (matrix, error) {
result, err := newMatrix(rows, cols)
if err != nil {
return nil, err
}
for r, row := range result {
for c := range row {
result[r][c] = galExp(byte(r), c)
}
}
return result, nil
}

217
vendor/github.com/klauspost/reedsolomon/matrix_test.go generated vendored
View File

@@ -1,217 +0,0 @@
/**
* Unit tests for Matrix
*
* Copyright 2015, Klaus Post
* Copyright 2015, Backblaze, Inc. All rights reserved.
*/
package reedsolomon
import (
"testing"
)
// TestNewMatrix - Tests validate the result for invalid input and the allocations made by newMatrix method.
func TestNewMatrix(t *testing.T) {
testCases := []struct {
rows int
columns int
// flag to indicate whether the test should pass.
shouldPass bool
expectedResult matrix
expectedErr error
}{
// Test case - 1.
// Test case with a negative row size.
{-1, 10, false, nil, errInvalidRowSize},
// Test case - 2.
// Test case with a negative column size.
{10, -1, false, nil, errInvalidColSize},
// Test case - 3.
// Test case with negative value for both row and column size.
{-1, -1, false, nil, errInvalidRowSize},
// Test case - 4.
// Test case with 0 value for row size.
{0, 10, false, nil, errInvalidRowSize},
// Test case - 5.
// Test case with 0 value for column size.
{-1, 0, false, nil, errInvalidRowSize},
// Test case - 6.
// Test case with 0 value for both row and column size.
{0, 0, false, nil, errInvalidRowSize},
}
for i, testCase := range testCases {
actualResult, actualErr := newMatrix(testCase.rows, testCase.columns)
if actualErr != nil && testCase.shouldPass {
t.Errorf("Test %d: Expected to pass, but failed with: <ERROR> %s", i+1, actualErr.Error())
}
if actualErr == nil && !testCase.shouldPass {
t.Errorf("Test %d: Expected to fail with <ERROR> \"%s\", but passed instead.", i+1, testCase.expectedErr)
}
// Failed as expected, but does it fail for the expected reason.
if actualErr != nil && !testCase.shouldPass {
if testCase.expectedErr != actualErr {
t.Errorf("Test %d: Expected to fail with error \"%s\", but instead failed with error \"%s\" instead.", i+1, testCase.expectedErr, actualErr)
}
}
// Test passes as expected, but the output values
// are verified for correctness here.
if actualErr == nil && testCase.shouldPass {
if testCase.rows != len(actualResult) {
// End the tests here if the the size doesn't match number of rows.
t.Fatalf("Test %d: Expected the size of the row of the new matrix to be `%d`, but instead found `%d`", i+1, testCase.rows, len(actualResult))
}
// Iterating over each row and validating the size of the column.
for j, row := range actualResult {
// If the row check passes, verify the size of each columns.
if testCase.columns != len(row) {
t.Errorf("Test %d: Row %d: Expected the size of the column of the new matrix to be `%d`, but instead found `%d`", i+1, j+1, testCase.columns, len(row))
}
}
}
}
}
// TestMatrixIdentity - validates the method for returning identity matrix of given size.
func TestMatrixIdentity(t *testing.T) {
m, err := identityMatrix(3)
if err != nil {
t.Fatal(err)
}
str := m.String()
expect := "[[1, 0, 0], [0, 1, 0], [0, 0, 1]]"
if str != expect {
t.Fatal(str, "!=", expect)
}
}
// Tests validate the output of matix multiplication method.
func TestMatrixMultiply(t *testing.T) {
m1, err := newMatrixData(
[][]byte{
[]byte{1, 2},
[]byte{3, 4},
})
if err != nil {
t.Fatal(err)
}
m2, err := newMatrixData(
[][]byte{
[]byte{5, 6},
[]byte{7, 8},
})
if err != nil {
t.Fatal(err)
}
actual, err := m1.Multiply(m2)
if err != nil {
t.Fatal(err)
}
str := actual.String()
expect := "[[11, 22], [19, 42]]"
if str != expect {
t.Fatal(str, "!=", expect)
}
}
// Tests validate the output of the method with computes inverse of matrix.
func TestMatrixInverse(t *testing.T) {
testCases := []struct {
matrixData [][]byte
// expected inverse matrix.
expectedResult string
// flag indicating whether the test should pass.
shouldPass bool
expectedErr error
}{
// Test case - 1.
// Test case validating inverse of the input Matrix.
{
// input data to construct the matrix.
[][]byte{
[]byte{56, 23, 98},
[]byte{3, 100, 200},
[]byte{45, 201, 123},
},
// expected Inverse matrix.
"[[175, 133, 33], [130, 13, 245], [112, 35, 126]]",
// test is expected to pass.
true,
nil,
},
// Test case - 2.
// Test case validating inverse of the input Matrix.
{
// input data to contruct the matrix.
[][]byte{
[]byte{1, 0, 0, 0, 0},
[]byte{0, 1, 0, 0, 0},
[]byte{0, 0, 0, 1, 0},
[]byte{0, 0, 0, 0, 1},
[]byte{7, 7, 6, 6, 1},
},
// expectedInverse matrix.
"[[1, 0, 0, 0, 0]," +
" [0, 1, 0, 0, 0]," +
" [123, 123, 1, 122, 122]," +
" [0, 0, 1, 0, 0]," +
" [0, 0, 0, 1, 0]]",
// test is expected to pass.
true,
nil,
},
// Test case with a non-square matrix.
// expected to fail with errNotSquare.
{
[][]byte{
[]byte{56, 23},
[]byte{3, 100},
[]byte{45, 201},
},
"",
false,
errNotSquare,
},
// Test case with singular matrix.
// expected to fail with error errSingular.
{
[][]byte{
[]byte{4, 2},
[]byte{12, 6},
},
"",
false,
errSingular,
},
}
for i, testCase := range testCases {
m, err := newMatrixData(testCase.matrixData)
if err != nil {
t.Fatalf("Test %d: Failed initializing new Matrix : %s", i+1, err)
}
actualResult, actualErr := m.Invert()
if actualErr != nil && testCase.shouldPass {
t.Errorf("Test %d: Expected to pass, but failed with: <ERROR> %s", i+1, actualErr.Error())
}
if actualErr == nil && !testCase.shouldPass {
t.Errorf("Test %d: Expected to fail with <ERROR> \"%s\", but passed instead.", i+1, testCase.expectedErr)
}
// Failed as expected, but does it fail for the expected reason.
if actualErr != nil && !testCase.shouldPass {
if testCase.expectedErr != actualErr {
t.Errorf("Test %d: Expected to fail with error \"%s\", but instead failed with error \"%s\" instead.", i+1, testCase.expectedErr, actualErr)
}
}
// Test passes as expected, but the output values
// are verified for correctness here.
if actualErr == nil && testCase.shouldPass {
if testCase.expectedResult != actualResult.String() {
t.Errorf("Test %d: The inverse matrix doesnt't match the expected result", i+1)
}
}
}
}

67
vendor/github.com/klauspost/reedsolomon/options.go generated vendored
View File

@@ -1,67 +0,0 @@
package reedsolomon
import (
"runtime"
"github.com/klauspost/cpuid"
)
// Option allows to override processing parameters.
type Option func(*options)
type options struct {
maxGoroutines int
minSplitSize int
useAVX2, useSSSE3 bool
}
var defaultOptions = options{
maxGoroutines: 50,
minSplitSize: 512,
}
func init() {
if runtime.GOMAXPROCS(0) <= 1 {
defaultOptions.maxGoroutines = 1
}
// Detect CPU capabilities.
defaultOptions.useSSSE3 = cpuid.CPU.SSSE3()
defaultOptions.useAVX2 = cpuid.CPU.AVX2()
}
// WithMaxGoroutines is the maximum number of goroutines number for encoding & decoding.
// Jobs will be split into this many parts, unless each goroutine would have to process
// less than minSplitSize bytes (set with WithMinSplitSize).
// For the best speed, keep this well above the GOMAXPROCS number for more fine grained
// scheduling.
// If n <= 0, it is ignored.
func WithMaxGoroutines(n int) Option {
return func(o *options) {
if n > 0 {
o.maxGoroutines = n
}
}
}
// MinSplitSize Is the minimum encoding size in bytes per goroutine.
// See WithMaxGoroutines on how jobs are split.
// If n <= 0, it is ignored.
func WithMinSplitSize(n int) Option {
return func(o *options) {
if n > 0 {
o.minSplitSize = n
}
}
}
func withSSE3(enabled bool) Option {
return func(o *options) {
o.useSSSE3 = enabled
}
}
func withAVX2(enabled bool) Option {
return func(o *options) {
o.useAVX2 = enabled
}
}

596
vendor/github.com/klauspost/reedsolomon/reedsolomon.go generated vendored
View File

@@ -1,596 +0,0 @@
/**
* Reed-Solomon Coding over 8-bit values.
*
* Copyright 2015, Klaus Post
* Copyright 2015, Backblaze, Inc.
*/
// Package reedsolomon enables Erasure Coding in Go
//
// For usage and examples, see https://github.com/klauspost/reedsolomon
//
package reedsolomon
import (
"bytes"
"errors"
"io"
"sync"
)
// Encoder is an interface to encode Reed-Salomon parity sets for your data.
type Encoder interface {
// Encodes parity for a set of data shards.
// Input is 'shards' containing data shards followed by parity shards.
// The number of shards must match the number given to New().
// Each shard is a byte array, and they must all be the same size.
// The parity shards will always be overwritten and the data shards
// will remain the same, so it is safe for you to read from the
// data shards while this is running.
Encode(shards [][]byte) error
// Verify returns true if the parity shards contain correct data.
// The data is the same format as Encode. No data is modified, so
// you are allowed to read from data while this is running.
Verify(shards [][]byte) (bool, error)
// Reconstruct will recreate the missing shards if possible.
//
// Given a list of shards, some of which contain data, fills in the
// ones that don't have data.
//
// The length of the array must be equal to the total number of shards.
// You indicate that a shard is missing by setting it to nil.
//
// If there are too few shards to reconstruct the missing
// ones, ErrTooFewShards will be returned.
//
// The reconstructed shard set is complete, but integrity is not verified.
// Use the Verify function to check if data set is ok.
Reconstruct(shards [][]byte) error
// Split a data slice into the number of shards given to the encoder,
// and create empty parity shards.
//
// The data will be split into equally sized shards.
// If the data size isn't dividable by the number of shards,
// the last shard will contain extra zeros.
//
// There must be at least 1 byte otherwise ErrShortData will be
// returned.
//
// The data will not be copied, except for the last shard, so you
// should not modify the data of the input slice afterwards.
Split(data []byte) ([][]byte, error)
// Join the shards and write the data segment to dst.
//
// Only the data shards are considered.
// You must supply the exact output size you want.
// If there are to few shards given, ErrTooFewShards will be returned.
// If the total data size is less than outSize, ErrShortData will be returned.
Join(dst io.Writer, shards [][]byte, outSize int) error
}
// reedSolomon contains a matrix for a specific
// distribution of datashards and parity shards.
// Construct if using New()
type reedSolomon struct {
DataShards int // Number of data shards, should not be modified.
ParityShards int // Number of parity shards, should not be modified.
Shards int // Total number of shards. Calculated, and should not be modified.
m matrix
tree inversionTree
parity [][]byte
o options
}
// ErrInvShardNum will be returned by New, if you attempt to create
// an Encoder where either data or parity shards is zero or less.
var ErrInvShardNum = errors.New("cannot create Encoder with zero or less data/parity shards")
// ErrMaxShardNum will be returned by New, if you attempt to create
// an Encoder where data and parity shards cannot be bigger than
// Galois field GF(2^8) - 1.
var ErrMaxShardNum = errors.New("cannot create Encoder with 255 or more data+parity shards")
// New creates a new encoder and initializes it to
// the number of data shards and parity shards that
// you want to use. You can reuse this encoder.
// Note that the maximum number of data shards is 256.
// If no options are supplied, default options are used.
func New(dataShards, parityShards int, opts ...Option) (Encoder, error) {
r := reedSolomon{
DataShards: dataShards,
ParityShards: parityShards,
Shards: dataShards + parityShards,
o: defaultOptions,
}
for _, opt := range opts {
opt(&r.o)
}
if dataShards <= 0 || parityShards <= 0 {
return nil, ErrInvShardNum
}
if dataShards+parityShards > 255 {
return nil, ErrMaxShardNum
}
// Start with a Vandermonde matrix. This matrix would work,
// in theory, but doesn't have the property that the data
// shards are unchanged after encoding.
vm, err := vandermonde(r.Shards, dataShards)
if err != nil {
return nil, err
}
// Multiply by the inverse of the top square of the matrix.
// This will make the top square be the identity matrix, but
// preserve the property that any square subset of rows is
// invertible.
top, _ := vm.SubMatrix(0, 0, dataShards, dataShards)
top, _ = top.Invert()
r.m, _ = vm.Multiply(top)
// Inverted matrices are cached in a tree keyed by the indices
// of the invalid rows of the data to reconstruct.
// The inversion root node will have the identity matrix as
// its inversion matrix because it implies there are no errors
// with the original data.
r.tree = newInversionTree(dataShards, parityShards)
r.parity = make([][]byte, parityShards)
for i := range r.parity {
r.parity[i] = r.m[dataShards+i]
}
return &r, err
}
// ErrTooFewShards is returned if too few shards where given to
// Encode/Verify/Reconstruct. It will also be returned from Reconstruct
// if there were too few shards to reconstruct the missing data.
var ErrTooFewShards = errors.New("too few shards given")
// Encodes parity for a set of data shards.
// An array 'shards' containing data shards followed by parity shards.
// The number of shards must match the number given to New.
// Each shard is a byte array, and they must all be the same size.
// The parity shards will always be overwritten and the data shards
// will remain the same.
func (r reedSolomon) Encode(shards [][]byte) error {
if len(shards) != r.Shards {
return ErrTooFewShards
}
err := checkShards(shards, false)
if err != nil {
return err
}
// Get the slice of output buffers.
output := shards[r.DataShards:]
// Do the coding.
r.codeSomeShards(r.parity, shards[0:r.DataShards], output, r.ParityShards, len(shards[0]))
return nil
}
// Verify returns true if the parity shards contain the right data.
// The data is the same format as Encode. No data is modified.
func (r reedSolomon) Verify(shards [][]byte) (bool, error) {
if len(shards) != r.Shards {
return false, ErrTooFewShards
}
err := checkShards(shards, false)
if err != nil {
return false, err
}
// Slice of buffers being checked.
toCheck := shards[r.DataShards:]
// Do the checking.
return r.checkSomeShards(r.parity, shards[0:r.DataShards], toCheck, r.ParityShards, len(shards[0])), nil
}
// Multiplies a subset of rows from a coding matrix by a full set of
// input shards to produce some output shards.
// 'matrixRows' is The rows from the matrix to use.
// 'inputs' An array of byte arrays, each of which is one input shard.
// The number of inputs used is determined by the length of each matrix row.
// outputs Byte arrays where the computed shards are stored.
// The number of outputs computed, and the
// number of matrix rows used, is determined by
// outputCount, which is the number of outputs to compute.
func (r reedSolomon) codeSomeShards(matrixRows, inputs, outputs [][]byte, outputCount, byteCount int) {
if r.o.maxGoroutines > 1 && byteCount > r.o.minSplitSize {
r.codeSomeShardsP(matrixRows, inputs, outputs, outputCount, byteCount)
return
}
for c := 0; c < r.DataShards; c++ {
in := inputs[c]
for iRow := 0; iRow < outputCount; iRow++ {
if c == 0 {
galMulSlice(matrixRows[iRow][c], in, outputs[iRow], r.o.useSSSE3, r.o.useAVX2)
} else {
galMulSliceXor(matrixRows[iRow][c], in, outputs[iRow], r.o.useSSSE3, r.o.useAVX2)
}
}
}
}
// Perform the same as codeSomeShards, but split the workload into
// several goroutines.
func (r reedSolomon) codeSomeShardsP(matrixRows, inputs, outputs [][]byte, outputCount, byteCount int) {
var wg sync.WaitGroup
do := byteCount / r.o.maxGoroutines
if do < r.o.minSplitSize {
do = r.o.minSplitSize
}
start := 0
for start < byteCount {
if start+do > byteCount {
do = byteCount - start
}
wg.Add(1)
go func(start, stop int) {
for c := 0; c < r.DataShards; c++ {
in := inputs[c]
for iRow := 0; iRow < outputCount; iRow++ {
if c == 0 {
galMulSlice(matrixRows[iRow][c], in[start:stop], outputs[iRow][start:stop], r.o.useSSSE3, r.o.useAVX2)
} else {
galMulSliceXor(matrixRows[iRow][c], in[start:stop], outputs[iRow][start:stop], r.o.useSSSE3, r.o.useAVX2)
}
}
}
wg.Done()
}(start, start+do)
start += do
}
wg.Wait()
}
// checkSomeShards is mostly the same as codeSomeShards,
// except this will check values and return
// as soon as a difference is found.
func (r reedSolomon) checkSomeShards(matrixRows, inputs, toCheck [][]byte, outputCount, byteCount int) bool {
if r.o.maxGoroutines > 1 && byteCount > r.o.minSplitSize {
return r.checkSomeShardsP(matrixRows, inputs, toCheck, outputCount, byteCount)
}
outputs := make([][]byte, len(toCheck))
for i := range outputs {
outputs[i] = make([]byte, byteCount)
}
for c := 0; c < r.DataShards; c++ {
in := inputs[c]
for iRow := 0; iRow < outputCount; iRow++ {
galMulSliceXor(matrixRows[iRow][c], in, outputs[iRow], r.o.useSSSE3, r.o.useAVX2)
}
}
for i, calc := range outputs {
if !bytes.Equal(calc, toCheck[i]) {
return false
}
}
return true
}
func (r reedSolomon) checkSomeShardsP(matrixRows, inputs, toCheck [][]byte, outputCount, byteCount int) bool {
same := true
var mu sync.RWMutex // For above
var wg sync.WaitGroup
do := byteCount / r.o.maxGoroutines
if do < r.o.minSplitSize {
do = r.o.minSplitSize
}
start := 0
for start < byteCount {
if start+do > byteCount {
do = byteCount - start
}
wg.Add(1)
go func(start, do int) {
defer wg.Done()
outputs := make([][]byte, len(toCheck))
for i := range outputs {
outputs[i] = make([]byte, do)
}
for c := 0; c < r.DataShards; c++ {
mu.RLock()
if !same {
mu.RUnlock()
return
}
mu.RUnlock()
in := inputs[c][start : start+do]
for iRow := 0; iRow < outputCount; iRow++ {
galMulSliceXor(matrixRows[iRow][c], in, outputs[iRow], r.o.useSSSE3, r.o.useAVX2)
}
}
for i, calc := range outputs {
if !bytes.Equal(calc, toCheck[i][start:start+do]) {
mu.Lock()
same = false
mu.Unlock()
return
}
}
}(start, do)
start += do
}
wg.Wait()
return same
}
// ErrShardNoData will be returned if there are no shards,
// or if the length of all shards is zero.
var ErrShardNoData = errors.New("no shard data")
// ErrShardSize is returned if shard length isn't the same for all
// shards.
var ErrShardSize = errors.New("shard sizes does not match")
// checkShards will check if shards are the same size
// or 0, if allowed. An error is returned if this fails.
// An error is also returned if all shards are size 0.
func checkShards(shards [][]byte, nilok bool) error {
size := shardSize(shards)
if size == 0 {
return ErrShardNoData
}
for _, shard := range shards {
if len(shard) != size {
if len(shard) != 0 || !nilok {
return ErrShardSize
}
}
}
return nil
}
// shardSize return the size of a single shard.
// The first non-zero size is returned,
// or 0 if all shards are size 0.
func shardSize(shards [][]byte) int {
for _, shard := range shards {
if len(shard) != 0 {
return len(shard)
}
}
return 0
}
// Reconstruct will recreate the missing shards, if possible.
//
// Given a list of shards, some of which contain data, fills in the
// ones that don't have data.
//
// The length of the array must be equal to Shards.
// You indicate that a shard is missing by setting it to nil.
//
// If there are too few shards to reconstruct the missing
// ones, ErrTooFewShards will be returned.
//
// The reconstructed shard set is complete, but integrity is not verified.
// Use the Verify function to check if data set is ok.
func (r reedSolomon) Reconstruct(shards [][]byte) error {
if len(shards) != r.Shards {
return ErrTooFewShards
}
// Check arguments.
err := checkShards(shards, true)
if err != nil {
return err
}
shardSize := shardSize(shards)
// Quick check: are all of the shards present? If so, there's
// nothing to do.
numberPresent := 0
for i := 0; i < r.Shards; i++ {
if len(shards[i]) != 0 {
numberPresent++
}
}
if numberPresent == r.Shards {
// Cool. All of the shards data data. We don't
// need to do anything.
return nil
}
// More complete sanity check
if numberPresent < r.DataShards {
return ErrTooFewShards
}
// Pull out an array holding just the shards that
// correspond to the rows of the submatrix. These shards
// will be the input to the decoding process that re-creates
// the missing data shards.
//
// Also, create an array of indices of the valid rows we do have
// and the invalid rows we don't have up until we have enough valid rows.
subShards := make([][]byte, r.DataShards)
validIndices := make([]int, r.DataShards)
invalidIndices := make([]int, 0)
subMatrixRow := 0
for matrixRow := 0; matrixRow < r.Shards && subMatrixRow < r.DataShards; matrixRow++ {
if len(shards[matrixRow]) != 0 {
subShards[subMatrixRow] = shards[matrixRow]
validIndices[subMatrixRow] = matrixRow
subMatrixRow++
} else {
invalidIndices = append(invalidIndices, matrixRow)
}
}
// Attempt to get the cached inverted matrix out of the tree
// based on the indices of the invalid rows.
dataDecodeMatrix := r.tree.GetInvertedMatrix(invalidIndices)
// If the inverted matrix isn't cached in the tree yet we must
// construct it ourselves and insert it into the tree for the
// future. In this way the inversion tree is lazily loaded.
if dataDecodeMatrix == nil {
// Pull out the rows of the matrix that correspond to the
// shards that we have and build a square matrix. This
// matrix could be used to generate the shards that we have
// from the original data.
subMatrix, _ := newMatrix(r.DataShards, r.DataShards)
for subMatrixRow, validIndex := range validIndices {
for c := 0; c < r.DataShards; c++ {
subMatrix[subMatrixRow][c] = r.m[validIndex][c]
}
}
// Invert the matrix, so we can go from the encoded shards
// back to the original data. Then pull out the row that
// generates the shard that we want to decode. Note that
// since this matrix maps back to the original data, it can
// be used to create a data shard, but not a parity shard.
dataDecodeMatrix, err = subMatrix.Invert()
if err != nil {
return err
}
// Cache the inverted matrix in the tree for future use keyed on the
// indices of the invalid rows.
err = r.tree.InsertInvertedMatrix(invalidIndices, dataDecodeMatrix, r.Shards)
if err != nil {
return err
}
}
// Re-create any data shards that were missing.
//
// The input to the coding is all of the shards we actually
// have, and the output is the missing data shards. The computation
// is done using the special decode matrix we just built.
outputs := make([][]byte, r.ParityShards)
matrixRows := make([][]byte, r.ParityShards)
outputCount := 0
for iShard := 0; iShard < r.DataShards; iShard++ {
if len(shards[iShard]) == 0 {
shards[iShard] = make([]byte, shardSize)
outputs[outputCount] = shards[iShard]
matrixRows[outputCount] = dataDecodeMatrix[iShard]
outputCount++
}
}
r.codeSomeShards(matrixRows, subShards, outputs[:outputCount], outputCount, shardSize)
// Now that we have all of the data shards intact, we can
// compute any of the parity that is missing.
//
// The input to the coding is ALL of the data shards, including
// any that we just calculated. The output is whichever of the
// data shards were missing.
outputCount = 0
for iShard := r.DataShards; iShard < r.Shards; iShard++ {
if len(shards[iShard]) == 0 {
shards[iShard] = make([]byte, shardSize)
outputs[outputCount] = shards[iShard]
matrixRows[outputCount] = r.parity[iShard-r.DataShards]
outputCount++
}
}
r.codeSomeShards(matrixRows, shards[:r.DataShards], outputs[:outputCount], outputCount, shardSize)
return nil
}
// ErrShortData will be returned by Split(), if there isn't enough data
// to fill the number of shards.
var ErrShortData = errors.New("not enough data to fill the number of requested shards")
// Split a data slice into the number of shards given to the encoder,
// and create empty parity shards.
//
// The data will be split into equally sized shards.
// If the data size isn't divisible by the number of shards,
// the last shard will contain extra zeros.
//
// There must be at least 1 byte otherwise ErrShortData will be
// returned.
//
// The data will not be copied, except for the last shard, so you
// should not modify the data of the input slice afterwards.
func (r reedSolomon) Split(data []byte) ([][]byte, error) {
if len(data) == 0 {
return nil, ErrShortData
}
// Calculate number of bytes per shard.
perShard := (len(data) + r.DataShards - 1) / r.DataShards
// Pad data to r.Shards*perShard.
padding := make([]byte, (r.Shards*perShard)-len(data))
data = append(data, padding...)
// Split into equal-length shards.
dst := make([][]byte, r.Shards)
for i := range dst {
dst[i] = data[:perShard]
data = data[perShard:]
}
return dst, nil
}
// ErrReconstructRequired is returned if too few data shards are intact and a
// reconstruction is required before you can successfully join the shards.
var ErrReconstructRequired = errors.New("reconstruction required as one or more required data shards are nil")
// Join the shards and write the data segment to dst.
//
// Only the data shards are considered.
// You must supply the exact output size you want.
//
// If there are to few shards given, ErrTooFewShards will be returned.
// If the total data size is less than outSize, ErrShortData will be returned.
// If one or more required data shards are nil, ErrReconstructRequired will be returned.
func (r reedSolomon) Join(dst io.Writer, shards [][]byte, outSize int) error {
// Do we have enough shards?
if len(shards) < r.DataShards {
return ErrTooFewShards
}
shards = shards[:r.DataShards]
// Do we have enough data?
size := 0
for _, shard := range shards {
if shard == nil {
return ErrReconstructRequired
}
size += len(shard)
// Do we have enough data already?
if size >= outSize {
break
}
}
if size < outSize {
return ErrShortData
}
// Copy data to dst
write := outSize
for _, shard := range shards {
if write < len(shard) {
_, err := dst.Write(shard[:write])
return err
}
n, err := dst.Write(shard)
if err != nil {
return err
}
write -= n
}
return nil
}

761
vendor/github.com/klauspost/reedsolomon/reedsolomon_test.go generated vendored
View File

@@ -1,761 +0,0 @@
/**
* Unit tests for ReedSolomon
*
* Copyright 2015, Klaus Post
* Copyright 2015, Backblaze, Inc. All rights reserved.
*/
package reedsolomon
import (
"bytes"
"math/rand"
"runtime"
"testing"
)
func testOpts() [][]Option {
if !testing.Short() {
return [][]Option{}
}
opts := [][]Option{
{WithMaxGoroutines(1), WithMinSplitSize(500), withSSE3(false), withAVX2(false)},
{WithMaxGoroutines(5000), WithMinSplitSize(50), withSSE3(false), withAVX2(false)},
{WithMaxGoroutines(5000), WithMinSplitSize(500000), withSSE3(false), withAVX2(false)},
{WithMaxGoroutines(1), WithMinSplitSize(500000), withSSE3(false), withAVX2(false)},
}
for _, o := range opts[:] {
if defaultOptions.useSSSE3 {
n := make([]Option, len(o), len(o)+1)
copy(n, o)
n = append(n, withSSE3(true))
opts = append(opts, n)
}
if defaultOptions.useAVX2 {
n := make([]Option, len(o), len(o)+1)
copy(n, o)
n = append(n, withAVX2(true))
opts = append(opts, n)
}
}
return opts
}
func TestEncoding(t *testing.T) {
testEncoding(t)
for _, o := range testOpts() {
testEncoding(t, o...)
}
}
func testEncoding(t *testing.T, o ...Option) {
perShard := 50000
r, err := New(10, 3, o...)
if err != nil {
t.Fatal(err)
}
shards := make([][]byte, 13)
for s := range shards {
shards[s] = make([]byte, perShard)
}
rand.Seed(0)
for s := 0; s < 13; s++ {
fillRandom(shards[s])
}
err = r.Encode(shards)
if err != nil {
t.Fatal(err)
}
ok, err := r.Verify(shards)
if err != nil {
t.Fatal(err)
}
if !ok {
t.Fatal("Verification failed")
}
err = r.Encode(make([][]byte, 1))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
badShards := make([][]byte, 13)
badShards[0] = make([]byte, 1)
err = r.Encode(badShards)
if err != ErrShardSize {
t.Errorf("expected %v, got %v", ErrShardSize, err)
}
}
func TestReconstruct(t *testing.T) {
testReconstruct(t)
for _, o := range testOpts() {
testReconstruct(t, o...)
}
}
func testReconstruct(t *testing.T, o ...Option) {
perShard := 50000
r, err := New(10, 3, o...)
if err != nil {
t.Fatal(err)
}
shards := make([][]byte, 13)
for s := range shards {
shards[s] = make([]byte, perShard)
}
rand.Seed(0)
for s := 0; s < 13; s++ {
fillRandom(shards[s])
}
err = r.Encode(shards)
if err != nil {
t.Fatal(err)
}
// Reconstruct with all shards present
err = r.Reconstruct(shards)
if err != nil {
t.Fatal(err)
}
// Reconstruct with 10 shards present
shards[0] = nil
shards[7] = nil
shards[11] = nil
err = r.Reconstruct(shards)
if err != nil {
t.Fatal(err)
}
ok, err := r.Verify(shards)
if err != nil {
t.Fatal(err)
}
if !ok {
t.Fatal("Verification failed")
}
// Reconstruct with 9 shards present (should fail)
shards[0] = nil
shards[4] = nil
shards[7] = nil
shards[11] = nil
err = r.Reconstruct(shards)
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = r.Reconstruct(make([][]byte, 1))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = r.Reconstruct(make([][]byte, 13))
if err != ErrShardNoData {
t.Errorf("expected %v, got %v", ErrShardNoData, err)
}
}
func TestVerify(t *testing.T) {
testVerify(t)
for _, o := range testOpts() {
testVerify(t, o...)
}
}
func testVerify(t *testing.T, o ...Option) {
perShard := 33333
r, err := New(10, 4, o...)
if err != nil {
t.Fatal(err)
}
shards := make([][]byte, 14)
for s := range shards {
shards[s] = make([]byte, perShard)
}
rand.Seed(0)
for s := 0; s < 10; s++ {
fillRandom(shards[s])
}
err = r.Encode(shards)
if err != nil {
t.Fatal(err)
}
ok, err := r.Verify(shards)
if err != nil {
t.Fatal(err)
}
if !ok {
t.Fatal("Verification failed")
}
// Put in random data. Verification should fail
fillRandom(shards[10])
ok, err = r.Verify(shards)
if err != nil {
t.Fatal(err)
}
if ok {
t.Fatal("Verification did not fail")
}
// Re-encode
err = r.Encode(shards)
if err != nil {
t.Fatal(err)
}
// Fill a data segment with random data
fillRandom(shards[0])
ok, err = r.Verify(shards)
if err != nil {
t.Fatal(err)
}
if ok {
t.Fatal("Verification did not fail")
}
_, err = r.Verify(make([][]byte, 1))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
_, err = r.Verify(make([][]byte, 14))
if err != ErrShardNoData {
t.Errorf("expected %v, got %v", ErrShardNoData, err)
}
}
func TestOneEncode(t *testing.T) {
codec, err := New(5, 5)
if err != nil {
t.Fatal(err)
}
shards := [][]byte{
{0, 1},
{4, 5},
{2, 3},
{6, 7},
{8, 9},
{0, 0},
{0, 0},
{0, 0},
{0, 0},
{0, 0},
}
codec.Encode(shards)
if shards[5][0] != 12 || shards[5][1] != 13 {
t.Fatal("shard 5 mismatch")
}
if shards[6][0] != 10 || shards[6][1] != 11 {
t.Fatal("shard 6 mismatch")
}
if shards[7][0] != 14 || shards[7][1] != 15 {
t.Fatal("shard 7 mismatch")
}
if shards[8][0] != 90 || shards[8][1] != 91 {
t.Fatal("shard 8 mismatch")
}
if shards[9][0] != 94 || shards[9][1] != 95 {
t.Fatal("shard 9 mismatch")
}
ok, err := codec.Verify(shards)
if err != nil {
t.Fatal(err)
}
if !ok {
t.Fatal("did not verify")
}
shards[8][0]++
ok, err = codec.Verify(shards)
if err != nil {
t.Fatal(err)
}
if ok {
t.Fatal("verify did not fail as expected")
}
}
func fillRandom(p []byte) {
for i := 0; i < len(p); i += 7 {
val := rand.Int63()
for j := 0; i+j < len(p) && j < 7; j++ {
p[i+j] = byte(val)
val >>= 8
}
}
}
func benchmarkEncode(b *testing.B, dataShards, parityShards, shardSize int) {
r, err := New(dataShards, parityShards)
if err != nil {
b.Fatal(err)
}
shards := make([][]byte, dataShards+parityShards)
for s := range shards {
shards[s] = make([]byte, shardSize)
}
rand.Seed(0)
for s := 0; s < dataShards; s++ {
fillRandom(shards[s])
}
b.SetBytes(int64(shardSize * dataShards))
b.ResetTimer()
for i := 0; i < b.N; i++ {
err = r.Encode(shards)
if err != nil {
b.Fatal(err)
}
}
}
func BenchmarkEncode10x2x10000(b *testing.B) {
benchmarkEncode(b, 10, 2, 10000)
}
func BenchmarkEncode100x20x10000(b *testing.B) {
benchmarkEncode(b, 100, 20, 10000)
}
func BenchmarkEncode17x3x1M(b *testing.B) {
benchmarkEncode(b, 17, 3, 1024*1024)
}
// Benchmark 10 data shards and 4 parity shards with 16MB each.
func BenchmarkEncode10x4x16M(b *testing.B) {
benchmarkEncode(b, 10, 4, 16*1024*1024)
}
// Benchmark 5 data shards and 2 parity shards with 1MB each.
func BenchmarkEncode5x2x1M(b *testing.B) {
benchmarkEncode(b, 5, 2, 1024*1024)
}
// Benchmark 1 data shards and 2 parity shards with 1MB each.
func BenchmarkEncode10x2x1M(b *testing.B) {
benchmarkEncode(b, 10, 2, 1024*1024)
}
// Benchmark 10 data shards and 4 parity shards with 1MB each.
func BenchmarkEncode10x4x1M(b *testing.B) {
benchmarkEncode(b, 10, 4, 1024*1024)
}
// Benchmark 50 data shards and 20 parity shards with 1MB each.
func BenchmarkEncode50x20x1M(b *testing.B) {
benchmarkEncode(b, 50, 20, 1024*1024)
}
// Benchmark 17 data shards and 3 parity shards with 16MB each.
func BenchmarkEncode17x3x16M(b *testing.B) {
benchmarkEncode(b, 17, 3, 16*1024*1024)
}
func benchmarkVerify(b *testing.B, dataShards, parityShards, shardSize int) {
r, err := New(dataShards, parityShards)
if err != nil {
b.Fatal(err)
}
shards := make([][]byte, parityShards+dataShards)
for s := range shards {
shards[s] = make([]byte, shardSize)
}
rand.Seed(0)
for s := 0; s < dataShards; s++ {
fillRandom(shards[s])
}
err = r.Encode(shards)
if err != nil {
b.Fatal(err)
}
b.SetBytes(int64(shardSize * dataShards))
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err = r.Verify(shards)
if err != nil {
b.Fatal(err)
}
}
}
// Benchmark 10 data slices with 2 parity slices holding 10000 bytes each
func BenchmarkVerify10x2x10000(b *testing.B) {
benchmarkVerify(b, 10, 2, 10000)
}
// Benchmark 50 data slices with 5 parity slices holding 100000 bytes each
func BenchmarkVerify50x5x50000(b *testing.B) {
benchmarkVerify(b, 50, 5, 100000)
}
// Benchmark 10 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkVerify10x2x1M(b *testing.B) {
benchmarkVerify(b, 10, 2, 1024*1024)
}
// Benchmark 5 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkVerify5x2x1M(b *testing.B) {
benchmarkVerify(b, 5, 2, 1024*1024)
}
// Benchmark 10 data slices with 4 parity slices holding 1MB bytes each
func BenchmarkVerify10x4x1M(b *testing.B) {
benchmarkVerify(b, 10, 4, 1024*1024)
}
// Benchmark 5 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkVerify50x20x1M(b *testing.B) {
benchmarkVerify(b, 50, 20, 1024*1024)
}
// Benchmark 10 data slices with 4 parity slices holding 16MB bytes each
func BenchmarkVerify10x4x16M(b *testing.B) {
benchmarkVerify(b, 10, 4, 16*1024*1024)
}
func corruptRandom(shards [][]byte, dataShards, parityShards int) {
shardsToCorrupt := rand.Intn(parityShards)
for i := 1; i <= shardsToCorrupt; i++ {
shards[rand.Intn(dataShards+parityShards)] = nil
}
}
func benchmarkReconstruct(b *testing.B, dataShards, parityShards, shardSize int) {
r, err := New(dataShards, parityShards)
if err != nil {
b.Fatal(err)
}
shards := make([][]byte, parityShards+dataShards)
for s := range shards {
shards[s] = make([]byte, shardSize)
}
rand.Seed(0)
for s := 0; s < dataShards; s++ {
fillRandom(shards[s])
}
err = r.Encode(shards)
if err != nil {
b.Fatal(err)
}
b.SetBytes(int64(shardSize * dataShards))
b.ResetTimer()
for i := 0; i < b.N; i++ {
corruptRandom(shards, dataShards, parityShards)
err = r.Reconstruct(shards)
if err != nil {
b.Fatal(err)
}
ok, err := r.Verify(shards)
if err != nil {
b.Fatal(err)
}
if !ok {
b.Fatal("Verification failed")
}
}
}
// Benchmark 10 data slices with 2 parity slices holding 10000 bytes each
func BenchmarkReconstruct10x2x10000(b *testing.B) {
benchmarkReconstruct(b, 10, 2, 10000)
}
// Benchmark 50 data slices with 5 parity slices holding 100000 bytes each
func BenchmarkReconstruct50x5x50000(b *testing.B) {
benchmarkReconstruct(b, 50, 5, 100000)
}
// Benchmark 10 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkReconstruct10x2x1M(b *testing.B) {
benchmarkReconstruct(b, 10, 2, 1024*1024)
}
// Benchmark 5 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkReconstruct5x2x1M(b *testing.B) {
benchmarkReconstruct(b, 5, 2, 1024*1024)
}
// Benchmark 10 data slices with 4 parity slices holding 1MB bytes each
func BenchmarkReconstruct10x4x1M(b *testing.B) {
benchmarkReconstruct(b, 10, 4, 1024*1024)
}
// Benchmark 5 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkReconstruct50x20x1M(b *testing.B) {
benchmarkReconstruct(b, 50, 20, 1024*1024)
}
// Benchmark 10 data slices with 4 parity slices holding 16MB bytes each
func BenchmarkReconstruct10x4x16M(b *testing.B) {
benchmarkReconstruct(b, 10, 4, 16*1024*1024)
}
func benchmarkReconstructP(b *testing.B, dataShards, parityShards, shardSize int) {
r, err := New(dataShards, parityShards)
if err != nil {
b.Fatal(err)
}
b.SetBytes(int64(shardSize * dataShards))
runtime.GOMAXPROCS(runtime.NumCPU())
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
shards := make([][]byte, parityShards+dataShards)
for s := range shards {
shards[s] = make([]byte, shardSize)
}
rand.Seed(0)
for s := 0; s < dataShards; s++ {
fillRandom(shards[s])
}
err = r.Encode(shards)
if err != nil {
b.Fatal(err)
}
for pb.Next() {
corruptRandom(shards, dataShards, parityShards)
err = r.Reconstruct(shards)
if err != nil {
b.Fatal(err)
}
ok, err := r.Verify(shards)
if err != nil {
b.Fatal(err)
}
if !ok {
b.Fatal("Verification failed")
}
}
})
}
// Benchmark 10 data slices with 2 parity slices holding 10000 bytes each
func BenchmarkReconstructP10x2x10000(b *testing.B) {
benchmarkReconstructP(b, 10, 2, 10000)
}
// Benchmark 50 data slices with 5 parity slices holding 100000 bytes each
func BenchmarkReconstructP50x5x50000(b *testing.B) {
benchmarkReconstructP(b, 50, 5, 100000)
}
// Benchmark 10 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkReconstructP10x2x1M(b *testing.B) {
benchmarkReconstructP(b, 10, 2, 1024*1024)
}
// Benchmark 5 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkReconstructP5x2x1M(b *testing.B) {
benchmarkReconstructP(b, 5, 2, 1024*1024)
}
// Benchmark 10 data slices with 4 parity slices holding 1MB bytes each
func BenchmarkReconstructP10x4x1M(b *testing.B) {
benchmarkReconstructP(b, 10, 4, 1024*1024)
}
// Benchmark 5 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkReconstructP50x20x1M(b *testing.B) {
benchmarkReconstructP(b, 50, 20, 1024*1024)
}
// Benchmark 10 data slices with 4 parity slices holding 16MB bytes each
func BenchmarkReconstructP10x4x16M(b *testing.B) {
benchmarkReconstructP(b, 10, 4, 16*1024*1024)
}
func TestEncoderReconstruct(t *testing.T) {
testEncoderReconstruct(t)
for _, o := range testOpts() {
testEncoderReconstruct(t, o...)
}
}
func testEncoderReconstruct(t *testing.T, o ...Option) {
// Create some sample data
var data = make([]byte, 250000)
fillRandom(data)
// Create 5 data slices of 50000 elements each
enc, err := New(5, 3, o...)
if err != nil {
t.Fatal(err)
}
shards, err := enc.Split(data)
if err != nil {
t.Fatal(err)
}
err = enc.Encode(shards)
if err != nil {
t.Fatal(err)
}
// Check that it verifies
ok, err := enc.Verify(shards)
if !ok || err != nil {
t.Fatal("not ok:", ok, "err:", err)
}
// Delete a shard
shards[0] = nil
// Should reconstruct
err = enc.Reconstruct(shards)
if err != nil {
t.Fatal(err)
}
// Check that it verifies
ok, err = enc.Verify(shards)
if !ok || err != nil {
t.Fatal("not ok:", ok, "err:", err)
}
// Recover original bytes
buf := new(bytes.Buffer)
err = enc.Join(buf, shards, len(data))
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(buf.Bytes(), data) {
t.Fatal("recovered bytes do not match")
}
// Corrupt a shard
shards[0] = nil
shards[1][0], shards[1][500] = 75, 75
// Should reconstruct (but with corrupted data)
err = enc.Reconstruct(shards)
if err != nil {
t.Fatal(err)
}
// Check that it verifies
ok, err = enc.Verify(shards)
if ok || err != nil {
t.Fatal("error or ok:", ok, "err:", err)
}
// Recovered data should not match original
buf.Reset()
err = enc.Join(buf, shards, len(data))
if err != nil {
t.Fatal(err)
}
if bytes.Equal(buf.Bytes(), data) {
t.Fatal("corrupted data matches original")
}
}
func TestSplitJoin(t *testing.T) {
var data = make([]byte, 250000)
rand.Seed(0)
fillRandom(data)
enc, _ := New(5, 3)
shards, err := enc.Split(data)
if err != nil {
t.Fatal(err)
}
_, err = enc.Split([]byte{})
if err != ErrShortData {
t.Errorf("expected %v, got %v", ErrShortData, err)
}
buf := new(bytes.Buffer)
err = enc.Join(buf, shards, 50)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(buf.Bytes(), data[:50]) {
t.Fatal("recovered data does match original")
}
err = enc.Join(buf, [][]byte{}, 0)
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = enc.Join(buf, shards, len(data)+1)
if err != ErrShortData {
t.Errorf("expected %v, got %v", ErrShortData, err)
}
shards[0] = nil
err = enc.Join(buf, shards, len(data))
if err != ErrReconstructRequired {
t.Errorf("expected %v, got %v", ErrReconstructRequired, err)
}
}
func TestCodeSomeShards(t *testing.T) {
var data = make([]byte, 250000)
fillRandom(data)
enc, _ := New(5, 3)
r := enc.(*reedSolomon) // need to access private methods
shards, _ := enc.Split(data)
old := runtime.GOMAXPROCS(1)
r.codeSomeShards(r.parity, shards[:r.DataShards], shards[r.DataShards:], r.ParityShards, len(shards[0]))
// hopefully more than 1 CPU
runtime.GOMAXPROCS(runtime.NumCPU())
r.codeSomeShards(r.parity, shards[:r.DataShards], shards[r.DataShards:], r.ParityShards, len(shards[0]))
// reset MAXPROCS, otherwise testing complains
runtime.GOMAXPROCS(old)
}
func TestAllMatrices(t *testing.T) {
t.Skip("Skipping slow matrix check")
for i := 1; i < 257; i++ {
_, err := New(i, i)
if err != nil {
t.Fatal("creating matrix size", i, i, ":", err)
}
}
}
func TestNew(t *testing.T) {
tests := []struct {
data, parity int
err error
}{
{127, 127, nil},
{256, 256, ErrMaxShardNum},
{0, 1, ErrInvShardNum},
{1, 0, ErrInvShardNum},
{257, 1, ErrMaxShardNum},
// overflow causes r.Shards to be negative
{256, int(^uint(0) >> 1), errInvalidRowSize},
}
for _, test := range tests {
_, err := New(test.data, test.parity)
if err != test.err {
t.Errorf("New(%v, %v): expected %v, got %v", test.data, test.parity, test.err, err)
}
}
}

575
vendor/github.com/klauspost/reedsolomon/streaming.go generated vendored
View File

@@ -1,575 +0,0 @@
/**
* Reed-Solomon Coding over 8-bit values.
*
* Copyright 2015, Klaus Post
* Copyright 2015, Backblaze, Inc.
*/
package reedsolomon
import (
"bytes"
"errors"
"fmt"
"io"
"sync"
)
// StreamEncoder is an interface to encode Reed-Salomon parity sets for your data.
// It provides a fully streaming interface, and processes data in blocks of up to 4MB.
//
// For small shard sizes, 10MB and below, it is recommended to use the in-memory interface,
// since the streaming interface has a start up overhead.
//
// For all operations, no readers and writers should not assume any order/size of
// individual reads/writes.
//
// For usage examples, see "stream-encoder.go" and "streamdecoder.go" in the examples
// folder.
type StreamEncoder interface {
// Encodes parity shards for a set of data shards.
//
// Input is 'shards' containing readers for data shards followed by parity shards
// io.Writer.
//
// The number of shards must match the number given to NewStream().
//
// Each reader must supply the same number of bytes.
//
// The parity shards will be written to the writer.
// The number of bytes written will match the input size.
//
// If a data stream returns an error, a StreamReadError type error
// will be returned. If a parity writer returns an error, a
// StreamWriteError will be returned.
Encode(data []io.Reader, parity []io.Writer) error
// Verify returns true if the parity shards contain correct data.
//
// The number of shards must match the number total data+parity shards
// given to NewStream().
//
// Each reader must supply the same number of bytes.
// If a shard stream returns an error, a StreamReadError type error
// will be returned.
Verify(shards []io.Reader) (bool, error)
// Reconstruct will recreate the missing shards if possible.
//
// Given a list of valid shards (to read) and invalid shards (to write)
//
// You indicate that a shard is missing by setting it to nil in the 'valid'
// slice and at the same time setting a non-nil writer in "fill".
// An index cannot contain both non-nil 'valid' and 'fill' entry.
// If both are provided 'ErrReconstructMismatch' is returned.
//
// If there are too few shards to reconstruct the missing
// ones, ErrTooFewShards will be returned.
//
// The reconstructed shard set is complete, but integrity is not verified.
// Use the Verify function to check if data set is ok.
Reconstruct(valid []io.Reader, fill []io.Writer) error
// Split a an input stream into the number of shards given to the encoder.
//
// The data will be split into equally sized shards.
// If the data size isn't dividable by the number of shards,
// the last shard will contain extra zeros.
//
// You must supply the total size of your input.
// 'ErrShortData' will be returned if it is unable to retrieve the
// number of bytes indicated.
Split(data io.Reader, dst []io.Writer, size int64) (err error)
// Join the shards and write the data segment to dst.
//
// Only the data shards are considered.
//
// You must supply the exact output size you want.
// If there are to few shards given, ErrTooFewShards will be returned.
// If the total data size is less than outSize, ErrShortData will be returned.
Join(dst io.Writer, shards []io.Reader, outSize int64) error
}
// StreamReadError is returned when a read error is encountered
// that relates to a supplied stream.
// This will allow you to find out which reader has failed.
type StreamReadError struct {
Err error // The error
Stream int // The stream number on which the error occurred
}
// Error returns the error as a string
func (s StreamReadError) Error() string {
return fmt.Sprintf("error reading stream %d: %s", s.Stream, s.Err)
}
// String returns the error as a string
func (s StreamReadError) String() string {
return s.Error()
}
// StreamWriteError is returned when a write error is encountered
// that relates to a supplied stream. This will allow you to
// find out which reader has failed.
type StreamWriteError struct {
Err error // The error
Stream int // The stream number on which the error occurred
}
// Error returns the error as a string
func (s StreamWriteError) Error() string {
return fmt.Sprintf("error writing stream %d: %s", s.Stream, s.Err)
}
// String returns the error as a string
func (s StreamWriteError) String() string {
return s.Error()
}
// rsStream contains a matrix for a specific
// distribution of datashards and parity shards.
// Construct if using NewStream()
type rsStream struct {
r *reedSolomon
bs int // Block size
// Shard reader
readShards func(dst [][]byte, in []io.Reader) error
// Shard writer
writeShards func(out []io.Writer, in [][]byte) error
creads bool
cwrites bool
}
// NewStream creates a new encoder and initializes it to
// the number of data shards and parity shards that
// you want to use. You can reuse this encoder.
// Note that the maximum number of data shards is 256.
func NewStream(dataShards, parityShards int, o ...Option) (StreamEncoder, error) {
enc, err := New(dataShards, parityShards, o...)
if err != nil {
return nil, err
}
rs := enc.(*reedSolomon)
r := rsStream{r: rs, bs: 4 << 20}
r.readShards = readShards
r.writeShards = writeShards
return &r, err
}
// NewStreamC creates a new encoder and initializes it to
// the number of data shards and parity shards given.
//
// This functions as 'NewStream', but allows you to enable CONCURRENT reads and writes.
func NewStreamC(dataShards, parityShards int, conReads, conWrites bool, o ...Option) (StreamEncoder, error) {
enc, err := New(dataShards, parityShards, o...)
if err != nil {
return nil, err
}
rs := enc.(*reedSolomon)
r := rsStream{r: rs, bs: 4 << 20}
r.readShards = readShards
r.writeShards = writeShards
if conReads {
r.readShards = cReadShards
}
if conWrites {
r.writeShards = cWriteShards
}
return &r, err
}
func createSlice(n, length int) [][]byte {
out := make([][]byte, n)
for i := range out {
out[i] = make([]byte, length)
}
return out
}
// Encodes parity shards for a set of data shards.
//
// Input is 'shards' containing readers for data shards followed by parity shards
// io.Writer.
//
// The number of shards must match the number given to NewStream().
//
// Each reader must supply the same number of bytes.
//
// The parity shards will be written to the writer.
// The number of bytes written will match the input size.
//
// If a data stream returns an error, a StreamReadError type error
// will be returned. If a parity writer returns an error, a
// StreamWriteError will be returned.
func (r rsStream) Encode(data []io.Reader, parity []io.Writer) error {
if len(data) != r.r.DataShards {
return ErrTooFewShards
}
if len(parity) != r.r.ParityShards {
return ErrTooFewShards
}
all := createSlice(r.r.Shards, r.bs)
in := all[:r.r.DataShards]
out := all[r.r.DataShards:]
read := 0
for {
err := r.readShards(in, data)
switch err {
case nil:
case io.EOF:
if read == 0 {
return ErrShardNoData
}
return nil
default:
return err
}
out = trimShards(out, shardSize(in))
read += shardSize(in)
err = r.r.Encode(all)
if err != nil {
return err
}
err = r.writeShards(parity, out)
if err != nil {
return err
}
}
}
// Trim the shards so they are all the same size
func trimShards(in [][]byte, size int) [][]byte {
for i := range in {
if in[i] != nil {
in[i] = in[i][0:size]
}
if len(in[i]) < size {
in[i] = nil
}
}
return in
}
func readShards(dst [][]byte, in []io.Reader) error {
if len(in) != len(dst) {
panic("internal error: in and dst size does not match")
}
size := -1
for i := range in {
if in[i] == nil {
dst[i] = nil
continue
}
n, err := io.ReadFull(in[i], dst[i])
// The error is EOF only if no bytes were read.
// If an EOF happens after reading some but not all the bytes,
// ReadFull returns ErrUnexpectedEOF.
switch err {
case io.ErrUnexpectedEOF, io.EOF:
if size < 0 {
size = n
} else if n != size {
// Shard sizes must match.
return ErrShardSize
}
dst[i] = dst[i][0:n]
case nil:
continue
default:
return StreamReadError{Err: err, Stream: i}
}
}
if size == 0 {
return io.EOF
}
return nil
}
func writeShards(out []io.Writer, in [][]byte) error {
if len(out) != len(in) {
panic("internal error: in and out size does not match")
}
for i := range in {
if out[i] == nil {
continue
}
n, err := out[i].Write(in[i])
if err != nil {
return StreamWriteError{Err: err, Stream: i}
}
//
if n != len(in[i]) {
return StreamWriteError{Err: io.ErrShortWrite, Stream: i}
}
}
return nil
}
type readResult struct {
n int
size int
err error
}
// cReadShards reads shards concurrently
func cReadShards(dst [][]byte, in []io.Reader) error {
if len(in) != len(dst) {
panic("internal error: in and dst size does not match")
}
var wg sync.WaitGroup
wg.Add(len(in))
res := make(chan readResult, len(in))
for i := range in {
if in[i] == nil {
dst[i] = nil
wg.Done()
continue
}
go func(i int) {
defer wg.Done()
n, err := io.ReadFull(in[i], dst[i])
// The error is EOF only if no bytes were read.
// If an EOF happens after reading some but not all the bytes,
// ReadFull returns ErrUnexpectedEOF.
res <- readResult{size: n, err: err, n: i}
}(i)
}
wg.Wait()
close(res)
size := -1
for r := range res {
switch r.err {
case io.ErrUnexpectedEOF, io.EOF:
if size < 0 {
size = r.size
} else if r.size != size {
// Shard sizes must match.
return ErrShardSize
}
dst[r.n] = dst[r.n][0:r.size]
case nil:
default:
return StreamReadError{Err: r.err, Stream: r.n}
}
}
if size == 0 {
return io.EOF
}
return nil
}
// cWriteShards writes shards concurrently
func cWriteShards(out []io.Writer, in [][]byte) error {
if len(out) != len(in) {
panic("internal error: in and out size does not match")
}
var errs = make(chan error, len(out))
var wg sync.WaitGroup
wg.Add(len(out))
for i := range in {
go func(i int) {
defer wg.Done()
if out[i] == nil {
errs <- nil
return
}
n, err := out[i].Write(in[i])
if err != nil {
errs <- StreamWriteError{Err: err, Stream: i}
return
}
if n != len(in[i]) {
errs <- StreamWriteError{Err: io.ErrShortWrite, Stream: i}
}
}(i)
}
wg.Wait()
close(errs)
for err := range errs {
if err != nil {
return err
}
}
return nil
}
// Verify returns true if the parity shards contain correct data.
//
// The number of shards must match the number total data+parity shards
// given to NewStream().
//
// Each reader must supply the same number of bytes.
// If a shard stream returns an error, a StreamReadError type error
// will be returned.
func (r rsStream) Verify(shards []io.Reader) (bool, error) {
if len(shards) != r.r.Shards {
return false, ErrTooFewShards
}
read := 0
all := createSlice(r.r.Shards, r.bs)
for {
err := r.readShards(all, shards)
if err == io.EOF {
if read == 0 {
return false, ErrShardNoData
}
return true, nil
}
if err != nil {
return false, err
}
read += shardSize(all)
ok, err := r.r.Verify(all)
if !ok || err != nil {
return ok, err
}
}
}
// ErrReconstructMismatch is returned by the StreamEncoder, if you supply
// "valid" and "fill" streams on the same index.
// Therefore it is impossible to see if you consider the shard valid
// or would like to have it reconstructed.
var ErrReconstructMismatch = errors.New("valid shards and fill shards are mutually exclusive")
// Reconstruct will recreate the missing shards if possible.
//
// Given a list of valid shards (to read) and invalid shards (to write)
//
// You indicate that a shard is missing by setting it to nil in the 'valid'
// slice and at the same time setting a non-nil writer in "fill".
// An index cannot contain both non-nil 'valid' and 'fill' entry.
//
// If there are too few shards to reconstruct the missing
// ones, ErrTooFewShards will be returned.
//
// The reconstructed shard set is complete, but integrity is not verified.
// Use the Verify function to check if data set is ok.
func (r rsStream) Reconstruct(valid []io.Reader, fill []io.Writer) error {
if len(valid) != r.r.Shards {
return ErrTooFewShards
}
if len(fill) != r.r.Shards {
return ErrTooFewShards
}
all := createSlice(r.r.Shards, r.bs)
for i := range valid {
if valid[i] != nil && fill[i] != nil {
return ErrReconstructMismatch
}
}
read := 0
for {
err := r.readShards(all, valid)
if err == io.EOF {
if read == 0 {
return ErrShardNoData
}
return nil
}
if err != nil {
return err
}
read += shardSize(all)
all = trimShards(all, shardSize(all))
err = r.r.Reconstruct(all)
if err != nil {
return err
}
err = r.writeShards(fill, all)
if err != nil {
return err
}
}
}
// Join the shards and write the data segment to dst.
//
// Only the data shards are considered.
//
// You must supply the exact output size you want.
// If there are to few shards given, ErrTooFewShards will be returned.
// If the total data size is less than outSize, ErrShortData will be returned.
func (r rsStream) Join(dst io.Writer, shards []io.Reader, outSize int64) error {
// Do we have enough shards?
if len(shards) < r.r.DataShards {
return ErrTooFewShards
}
// Trim off parity shards if any
shards = shards[:r.r.DataShards]
for i := range shards {
if shards[i] == nil {
return StreamReadError{Err: ErrShardNoData, Stream: i}
}
}
// Join all shards
src := io.MultiReader(shards...)
// Copy data to dst
n, err := io.CopyN(dst, src, outSize)
if err == io.EOF {
return ErrShortData
}
if err != nil {
return err
}
if n != outSize {
return ErrShortData
}
return nil
}
// Split a an input stream into the number of shards given to the encoder.
//
// The data will be split into equally sized shards.
// If the data size isn't dividable by the number of shards,
// the last shard will contain extra zeros.
//
// You must supply the total size of your input.
// 'ErrShortData' will be returned if it is unable to retrieve the
// number of bytes indicated.
func (r rsStream) Split(data io.Reader, dst []io.Writer, size int64) error {
if size == 0 {
return ErrShortData
}
if len(dst) != r.r.DataShards {
return ErrInvShardNum
}
for i := range dst {
if dst[i] == nil {
return StreamWriteError{Err: ErrShardNoData, Stream: i}
}
}
// Calculate number of bytes per shard.
perShard := (size + int64(r.r.DataShards) - 1) / int64(r.r.DataShards)
// Pad data to r.Shards*perShard.
padding := make([]byte, (int64(r.r.Shards)*perShard)-size)
data = io.MultiReader(data, bytes.NewBuffer(padding))
// Split into equal-length shards and copy.
for i := range dst {
n, err := io.CopyN(dst[i], data, perShard)
if err != io.EOF && err != nil {
return err
}
if n != perShard {
return ErrShortData
}
}
return nil
}

604
vendor/github.com/klauspost/reedsolomon/streaming_test.go generated vendored
View File

@@ -1,604 +0,0 @@
/**
* Unit tests for ReedSolomon Streaming API
*
* Copyright 2015, Klaus Post
*/
package reedsolomon
import (
"bytes"
"io"
"io/ioutil"
"math/rand"
"testing"
)
func TestStreamEncoding(t *testing.T) {
perShard := 10 << 20
if testing.Short() {
perShard = 50000
}
r, err := NewStream(10, 3)
if err != nil {
t.Fatal(err)
}
rand.Seed(0)
input := randomBytes(10, perShard)
data := toBuffers(input)
par := emptyBuffers(3)
err = r.Encode(toReaders(data), toWriters(par))
if err != nil {
t.Fatal(err)
}
// Reset Data
data = toBuffers(input)
all := append(toReaders(data), toReaders(par)...)
ok, err := r.Verify(all)
if err != nil {
t.Fatal(err)
}
if !ok {
t.Fatal("Verification failed")
}
err = r.Encode(toReaders(emptyBuffers(1)), toWriters(emptyBuffers(1)))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = r.Encode(toReaders(emptyBuffers(10)), toWriters(emptyBuffers(1)))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = r.Encode(toReaders(emptyBuffers(10)), toWriters(emptyBuffers(3)))
if err != ErrShardNoData {
t.Errorf("expected %v, got %v", ErrShardNoData, err)
}
badShards := emptyBuffers(10)
badShards[0] = randomBuffer(123)
err = r.Encode(toReaders(badShards), toWriters(emptyBuffers(3)))
if err != ErrShardSize {
t.Errorf("expected %v, got %v", ErrShardSize, err)
}
}
func TestStreamEncodingConcurrent(t *testing.T) {
perShard := 10 << 20
if testing.Short() {
perShard = 50000
}
r, err := NewStreamC(10, 3, true, true)
if err != nil {
t.Fatal(err)
}
rand.Seed(0)
input := randomBytes(10, perShard)
data := toBuffers(input)
par := emptyBuffers(3)
err = r.Encode(toReaders(data), toWriters(par))
if err != nil {
t.Fatal(err)
}
// Reset Data
data = toBuffers(input)
all := append(toReaders(data), toReaders(par)...)
ok, err := r.Verify(all)
if err != nil {
t.Fatal(err)
}
if !ok {
t.Fatal("Verification failed")
}
err = r.Encode(toReaders(emptyBuffers(1)), toWriters(emptyBuffers(1)))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = r.Encode(toReaders(emptyBuffers(10)), toWriters(emptyBuffers(1)))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = r.Encode(toReaders(emptyBuffers(10)), toWriters(emptyBuffers(3)))
if err != ErrShardNoData {
t.Errorf("expected %v, got %v", ErrShardNoData, err)
}
badShards := emptyBuffers(10)
badShards[0] = randomBuffer(123)
badShards[1] = randomBuffer(123)
err = r.Encode(toReaders(badShards), toWriters(emptyBuffers(3)))
if err != ErrShardSize {
t.Errorf("expected %v, got %v", ErrShardSize, err)
}
}
func randomBuffer(length int) *bytes.Buffer {
b := make([]byte, length)
fillRandom(b)
return bytes.NewBuffer(b)
}
func randomBytes(n, length int) [][]byte {
bufs := make([][]byte, n)
for j := range bufs {
bufs[j] = make([]byte, length)
fillRandom(bufs[j])
}
return bufs
}
func toBuffers(in [][]byte) []*bytes.Buffer {
out := make([]*bytes.Buffer, len(in))
for i := range in {
out[i] = bytes.NewBuffer(in[i])
}
return out
}
func toReaders(in []*bytes.Buffer) []io.Reader {
out := make([]io.Reader, len(in))
for i := range in {
out[i] = in[i]
}
return out
}
func toWriters(in []*bytes.Buffer) []io.Writer {
out := make([]io.Writer, len(in))
for i := range in {
out[i] = in[i]
}
return out
}
func nilWriters(n int) []io.Writer {
out := make([]io.Writer, n)
for i := range out {
out[i] = nil
}
return out
}
func emptyBuffers(n int) []*bytes.Buffer {
b := make([]*bytes.Buffer, n)
for i := range b {
b[i] = &bytes.Buffer{}
}
return b
}
func toBytes(in []*bytes.Buffer) [][]byte {
b := make([][]byte, len(in))
for i := range in {
b[i] = in[i].Bytes()
}
return b
}
func TestStreamReconstruct(t *testing.T) {
perShard := 10 << 20
if testing.Short() {
perShard = 50000
}
r, err := NewStream(10, 3)
if err != nil {
t.Fatal(err)
}
rand.Seed(0)
shards := randomBytes(10, perShard)
parb := emptyBuffers(3)
err = r.Encode(toReaders(toBuffers(shards)), toWriters(parb))
if err != nil {
t.Fatal(err)
}
parity := toBytes(parb)
all := append(toReaders(toBuffers(shards)), toReaders(toBuffers(parity))...)
fill := make([]io.Writer, 13)
// Reconstruct with all shards present, all fill nil
err = r.Reconstruct(all, fill)
if err != nil {
t.Fatal(err)
}
all = append(toReaders(toBuffers(shards)), toReaders(toBuffers(parity))...)
// Reconstruct with 10 shards present
all[0] = nil
fill[0] = emptyBuffers(1)[0]
all[7] = nil
fill[7] = emptyBuffers(1)[0]
all[11] = nil
fill[11] = emptyBuffers(1)[0]
err = r.Reconstruct(all, fill)
if err != nil {
t.Fatal(err)
}
shards[0] = fill[0].(*bytes.Buffer).Bytes()
shards[7] = fill[7].(*bytes.Buffer).Bytes()
parity[1] = fill[11].(*bytes.Buffer).Bytes()
all = append(toReaders(toBuffers(shards)), toReaders(toBuffers(parity))...)
ok, err := r.Verify(all)
if err != nil {
t.Fatal(err)
}
if !ok {
t.Fatal("Verification failed")
}
all = append(toReaders(toBuffers(shards)), toReaders(toBuffers(parity))...)
// Reconstruct with 9 shards present (should fail)
all[0] = nil
fill[0] = emptyBuffers(1)[0]
all[4] = nil
fill[4] = emptyBuffers(1)[0]
all[7] = nil
fill[7] = emptyBuffers(1)[0]
all[11] = nil
fill[11] = emptyBuffers(1)[0]
err = r.Reconstruct(all, fill)
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = r.Reconstruct(toReaders(emptyBuffers(3)), toWriters(emptyBuffers(3)))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = r.Reconstruct(toReaders(emptyBuffers(13)), toWriters(emptyBuffers(3)))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
err = r.Reconstruct(toReaders(emptyBuffers(13)), toWriters(emptyBuffers(13)))
if err != ErrReconstructMismatch {
t.Errorf("expected %v, got %v", ErrReconstructMismatch, err)
}
err = r.Reconstruct(toReaders(emptyBuffers(13)), nilWriters(13))
if err != ErrShardNoData {
t.Errorf("expected %v, got %v", ErrShardNoData, err)
}
}
func TestStreamVerify(t *testing.T) {
perShard := 10 << 20
if testing.Short() {
perShard = 50000
}
r, err := NewStream(10, 4)
if err != nil {
t.Fatal(err)
}
shards := randomBytes(10, perShard)
parb := emptyBuffers(4)
err = r.Encode(toReaders(toBuffers(shards)), toWriters(parb))
if err != nil {
t.Fatal(err)
}
parity := toBytes(parb)
all := append(toReaders(toBuffers(shards)), toReaders(parb)...)
ok, err := r.Verify(all)
if err != nil {
t.Fatal(err)
}
if !ok {
t.Fatal("Verification failed")
}
// Flip bits in a random byte
parity[0][len(parity[0])-20000] = parity[0][len(parity[0])-20000] ^ 0xff
all = append(toReaders(toBuffers(shards)), toReaders(toBuffers(parity))...)
ok, err = r.Verify(all)
if err != nil {
t.Fatal(err)
}
if ok {
t.Fatal("Verification did not fail")
}
// Re-encode
err = r.Encode(toReaders(toBuffers(shards)), toWriters(parb))
if err != nil {
t.Fatal(err)
}
// Fill a data segment with random data
shards[0][len(shards[0])-30000] = shards[0][len(shards[0])-30000] ^ 0xff
all = append(toReaders(toBuffers(shards)), toReaders(parb)...)
ok, err = r.Verify(all)
if err != nil {
t.Fatal(err)
}
if ok {
t.Fatal("Verification did not fail")
}
_, err = r.Verify(toReaders(emptyBuffers(10)))
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
_, err = r.Verify(toReaders(emptyBuffers(14)))
if err != ErrShardNoData {
t.Errorf("expected %v, got %v", ErrShardNoData, err)
}
}
func TestStreamOneEncode(t *testing.T) {
codec, err := NewStream(5, 5)
if err != nil {
t.Fatal(err)
}
shards := [][]byte{
{0, 1},
{4, 5},
{2, 3},
{6, 7},
{8, 9},
}
parb := emptyBuffers(5)
codec.Encode(toReaders(toBuffers(shards)), toWriters(parb))
parity := toBytes(parb)
if parity[0][0] != 12 || parity[0][1] != 13 {
t.Fatal("shard 5 mismatch")
}
if parity[1][0] != 10 || parity[1][1] != 11 {
t.Fatal("shard 6 mismatch")
}
if parity[2][0] != 14 || parity[2][1] != 15 {
t.Fatal("shard 7 mismatch")
}
if parity[3][0] != 90 || parity[3][1] != 91 {
t.Fatal("shard 8 mismatch")
}
if parity[4][0] != 94 || parity[4][1] != 95 {
t.Fatal("shard 9 mismatch")
}
all := append(toReaders(toBuffers(shards)), toReaders(toBuffers(parity))...)
ok, err := codec.Verify(all)
if err != nil {
t.Fatal(err)
}
if !ok {
t.Fatal("did not verify")
}
shards[3][0]++
all = append(toReaders(toBuffers(shards)), toReaders(toBuffers(parity))...)
ok, err = codec.Verify(all)
if err != nil {
t.Fatal(err)
}
if ok {
t.Fatal("verify did not fail as expected")
}
}
func benchmarkStreamEncode(b *testing.B, dataShards, parityShards, shardSize int) {
r, err := NewStream(dataShards, parityShards)
if err != nil {
b.Fatal(err)
}
shards := make([][]byte, dataShards)
for s := range shards {
shards[s] = make([]byte, shardSize)
}
rand.Seed(0)
for s := 0; s < dataShards; s++ {
fillRandom(shards[s])
}
b.SetBytes(int64(shardSize * dataShards))
b.ResetTimer()
out := make([]io.Writer, parityShards)
for i := range out {
out[i] = ioutil.Discard
}
for i := 0; i < b.N; i++ {
err = r.Encode(toReaders(toBuffers(shards)), out)
if err != nil {
b.Fatal(err)
}
}
}
func BenchmarkStreamEncode10x2x10000(b *testing.B) {
benchmarkStreamEncode(b, 10, 2, 10000)
}
func BenchmarkStreamEncode100x20x10000(b *testing.B) {
benchmarkStreamEncode(b, 100, 20, 10000)
}
func BenchmarkStreamEncode17x3x1M(b *testing.B) {
benchmarkStreamEncode(b, 17, 3, 1024*1024)
}
// Benchmark 10 data shards and 4 parity shards with 16MB each.
func BenchmarkStreamEncode10x4x16M(b *testing.B) {
benchmarkStreamEncode(b, 10, 4, 16*1024*1024)
}
// Benchmark 5 data shards and 2 parity shards with 1MB each.
func BenchmarkStreamEncode5x2x1M(b *testing.B) {
benchmarkStreamEncode(b, 5, 2, 1024*1024)
}
// Benchmark 1 data shards and 2 parity shards with 1MB each.
func BenchmarkStreamEncode10x2x1M(b *testing.B) {
benchmarkStreamEncode(b, 10, 2, 1024*1024)
}
// Benchmark 10 data shards and 4 parity shards with 1MB each.
func BenchmarkStreamEncode10x4x1M(b *testing.B) {
benchmarkStreamEncode(b, 10, 4, 1024*1024)
}
// Benchmark 50 data shards and 20 parity shards with 1MB each.
func BenchmarkStreamEncode50x20x1M(b *testing.B) {
benchmarkStreamEncode(b, 50, 20, 1024*1024)
}
// Benchmark 17 data shards and 3 parity shards with 16MB each.
func BenchmarkStreamEncode17x3x16M(b *testing.B) {
benchmarkStreamEncode(b, 17, 3, 16*1024*1024)
}
func benchmarkStreamVerify(b *testing.B, dataShards, parityShards, shardSize int) {
r, err := NewStream(dataShards, parityShards)
if err != nil {
b.Fatal(err)
}
shards := make([][]byte, parityShards+dataShards)
for s := range shards {
shards[s] = make([]byte, shardSize)
}
rand.Seed(0)
for s := 0; s < dataShards; s++ {
fillRandom(shards[s])
}
err = r.Encode(toReaders(toBuffers(shards[:dataShards])), toWriters(toBuffers(shards[dataShards:])))
if err != nil {
b.Fatal(err)
}
b.SetBytes(int64(shardSize * dataShards))
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err = r.Verify(toReaders(toBuffers(shards)))
if err != nil {
b.Fatal(err)
}
}
}
// Benchmark 10 data slices with 2 parity slices holding 10000 bytes each
func BenchmarkStreamVerify10x2x10000(b *testing.B) {
benchmarkStreamVerify(b, 10, 2, 10000)
}
// Benchmark 50 data slices with 5 parity slices holding 100000 bytes each
func BenchmarkStreamVerify50x5x50000(b *testing.B) {
benchmarkStreamVerify(b, 50, 5, 100000)
}
// Benchmark 10 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkStreamVerify10x2x1M(b *testing.B) {
benchmarkStreamVerify(b, 10, 2, 1024*1024)
}
// Benchmark 5 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkStreamVerify5x2x1M(b *testing.B) {
benchmarkStreamVerify(b, 5, 2, 1024*1024)
}
// Benchmark 10 data slices with 4 parity slices holding 1MB bytes each
func BenchmarkStreamVerify10x4x1M(b *testing.B) {
benchmarkStreamVerify(b, 10, 4, 1024*1024)
}
// Benchmark 5 data slices with 2 parity slices holding 1MB bytes each
func BenchmarkStreamVerify50x20x1M(b *testing.B) {
benchmarkStreamVerify(b, 50, 20, 1024*1024)
}
// Benchmark 10 data slices with 4 parity slices holding 16MB bytes each
func BenchmarkStreamVerify10x4x16M(b *testing.B) {
benchmarkStreamVerify(b, 10, 4, 16*1024*1024)
}
func TestStreamSplitJoin(t *testing.T) {
var data = make([]byte, 250000)
rand.Seed(0)
fillRandom(data)
enc, _ := NewStream(5, 3)
split := emptyBuffers(5)
err := enc.Split(bytes.NewBuffer(data), toWriters(split), int64(len(data)))
if err != nil {
t.Fatal(err)
}
splits := toBytes(split)
expect := len(data) / 5
// Beware, if changing data size
if split[0].Len() != expect {
t.Errorf("unexpected size. expected %d, got %d", expect, split[0].Len())
}
err = enc.Split(bytes.NewBuffer([]byte{}), toWriters(emptyBuffers(3)), 0)
if err != ErrShortData {
t.Errorf("expected %v, got %v", ErrShortData, err)
}
buf := new(bytes.Buffer)
err = enc.Join(buf, toReaders(toBuffers(splits)), int64(len(data)))
if err != nil {
t.Fatal(err)
}
joined := buf.Bytes()
if !bytes.Equal(joined, data) {
t.Fatal("recovered data does match original", joined[:8], data[:8], "... lengths:", len(joined), len(data))
}
err = enc.Join(buf, toReaders(emptyBuffers(2)), 0)
if err != ErrTooFewShards {
t.Errorf("expected %v, got %v", ErrTooFewShards, err)
}
bufs := toReaders(emptyBuffers(5))
bufs[2] = nil
err = enc.Join(buf, bufs, 0)
if se, ok := err.(StreamReadError); ok {
if se.Err != ErrShardNoData {
t.Errorf("expected %v, got %v", ErrShardNoData, se.Err)
}
if se.Stream != 2 {
t.Errorf("Expected error on stream 2, got %d", se.Stream)
}
} else {
t.Errorf("expected error type %T, got %T", StreamReadError{}, err)
}
err = enc.Join(buf, toReaders(toBuffers(splits)), int64(len(data)+1))
if err != ErrShortData {
t.Errorf("expected %v, got %v", ErrShortData, err)
}
}
func TestNewStream(t *testing.T) {
tests := []struct {
data, parity int
err error
}{
{127, 127, nil},
{256, 256, ErrMaxShardNum},
{0, 1, ErrInvShardNum},
{1, 0, ErrInvShardNum},
{257, 1, ErrMaxShardNum},
// overflow causes r.Shards to be negative
{256, int(^uint(0) >> 1), errInvalidRowSize},
}
for _, test := range tests {
_, err := NewStream(test.data, test.parity)
if err != test.err {
t.Errorf("New(%v, %v): expected %v, got %v", test.data, test.parity, test.err, err)
}
}
}