关于python：通过Cython将C向量传递给Numpy而不自动复制并负责内存管理


Passing C++ vector to Numpy through Cython without copying and taking care of memory management automatically

处理大型矩阵(NxM，1K <= N <= 20K)

相关讨论

我认为@FlorianWeimer的答案提供了一个不错的解决方案(分配一个vector并将其传递到您的C函数中)，但是应该可以从doit返回一个向量，并使用move构造函数避免复制。 >

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
from libcpp.vector cimport vector

cdef extern from"<utility>" namespace"std" nogil:
  T move[T](T) # don't worry that this doesn't quite match the c++ signature

cdef extern from"fast.h":
    vector[int] doit(int length)

# define ArrayWrapper as holding in a vector
cdef class ArrayWrapper:
    cdef vector[int] vec
    cdef Py_ssize_t shape[1]
    cdef Py_ssize_t strides[1]

    # constructor and destructor are fairly unimportant now since
    # vec will be destroyed automatically.

    cdef set_data(self, vector[int]& data):
       self.vec = move(data)
       # @ead suggests `self.vec.swap(data)` instead
       # to avoid having to wrap move

    # now implement the buffer protocol for the class
    # which makes it generally useful to anything that expects an array
    def __getbuffer__(self, Py_buffer *buffer, int flags):
        # relevant documentation http://cython.readthedocs.io/en/latest/src/userguide/buffer.html#a-matrix-class
        cdef Py_ssize_t itemsize = sizeof(self.vec[0])

        self.shape[0] = self.vec.size()
        self.strides[0] = sizeof(int)
        buffer.buf = <char *>&(self.vec[0])
        buffer.format = 'i'
        buffer.internal = NULL
        buffer.itemsize = itemsize
        buffer.len = self.v.size() * itemsize   # product(shape) * itemsize
        buffer.ndim = 1
        buffer.obj = self
        buffer.readonly = 0
        buffer.shape = self.shape
        buffer.strides = self.strides
        buffer.suboffsets = NULL

然后您应该可以将其用作：

1
2
3
4
cdef vector[int] array = doit(length)
cdef ArrayWrapper w
w.set_data(array) #"array" itself is invalid from here on
numpy_array = np.asarray(w)

编辑：Cython在C模板方面不是很好-它坚持编写std::move<vector<int>>(...)而不是std::move(...)然后让C推导类型。有时这会导致std::move问题。如果您对此有疑问，那么最好的解决方案通常是只告诉Cython您想要的重载：

1
2
 cdef extern from"<utility>" namespace"std" nogil:
    vector[int] move(vector[int])
相关讨论

从doit返回时，WhyNot对象超出范围，并且数组元素被释放。这意味着&WhyNot[0]不再是有效的指针。您需要将WhyNot对象存储在其他地方，可能存储在调用方提供的位置。

一种实现方法是将doit拆分为三个函数，doit_allocate分配向量并返回指向它的指针，doit像以前一样(但是带有一个参数，该参数接收指向预分配向量的指针) , and doit_free`，用于取消分配向量。

类似这样的东西：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
vector<int> *
doit_allocate()
{
    return new vector<int>;
}

int *
doit(vector<int> *WhyNot, int length)
{
    // Something really heavy
    cout <<"C++: doing it fast" << endl;

    // Heavy stuff - like reading a big file and preprocessing it
    for(int i=0; i<length; ++i)
        WhyNot->push_back(i); // heavy stuff

    cout <<"C++: did it really fast" << endl;
    return WhyNot->front();
}

void
doit_free(vector<int> *WhyNot)
{
    delete WhyNot;
}
相关讨论