git-lng-odp

git-lng-odp@lists.linaro.org

1 participants
1241 discussions

[COMMIT] branch master updated. v1.10.1.0-113-g032915e

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 032915e5988d0e1da0562193b8bc925600cdb5f6 (commit) from da7a72e86db0cf83f7b391d8516f9b7504f4e27a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 032915e5988d0e1da0562193b8bc925600cdb5f6 Author: Juha Mattila <juha.mattila(a)nokia.com> Date: Tue Sep 6 16:22:04 2016 +0300 linux-gen: tap: add classifier support Add missing classifier support to tap pktio. Resolves bug: https://bugs.linaro.org/show_bug.cgi?id=2504 Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Matias Elo <matias.elo(a)nokia.com> Signed-off-by: Juha Mattila <juha.mattila(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/pktio/tap.c b/platform/linux-generic/pktio/tap.c index d758a39..ac20456 100644 --- a/platform/linux-generic/pktio/tap.c +++ b/platform/linux-generic/pktio/tap.c @@ -43,6 +43,7 @@ #include <odp_packet_socket.h> #include <odp_packet_internal.h> #include <odp_packet_io_internal.h> +#include <odp_classification_internal.h> #define BUF_SIZE 65536 @@ -185,8 +186,17 @@ static odp_packet_t pack_odp_pkt(pktio_entry_t *pktio_entry, const void *data, { odp_packet_t pkt; odp_packet_hdr_t *pkt_hdr; + odp_packet_hdr_t parsed_hdr; int num; + if (pktio_cls_enabled(pktio_entry)) { + if (cls_classify_packet(pktio_entry, data, len, len, + &pktio_entry->s.pkt_tap.pool, + &parsed_hdr)) { + return ODP_PACKET_INVALID; + } + } + num = packet_alloc_multi(pktio_entry->s.pkt_tap.pool, len, &pkt, 1); if (num != 1) @@ -199,7 +209,12 @@ static odp_packet_t pack_odp_pkt(pktio_entry_t *pktio_entry, const void *data, } pkt_hdr = odp_packet_hdr(pkt); - packet_parse_l2(&pkt_hdr->p, len); + + if (pktio_cls_enabled(pktio_entry)) + copy_packet_cls_metadata(&parsed_hdr, pkt_hdr); + else + packet_parse_l2(&pkt_hdr->p, len); + packet_set_ts(pkt_hdr, ts); pkt_hdr->input = pktio_entry->s.handle; ----------------------------------------------------------------------- Summary of changes: platform/linux-generic/pktio/tap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) hooks/post-receive --

10 years

[COMMIT] branch master updated. v1.10.1.0-112-gda7a72e

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via da7a72e86db0cf83f7b391d8516f9b7504f4e27a (commit) from a7f5589f81864c71faf18c3d2c3636320cecb1c6 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit da7a72e86db0cf83f7b391d8516f9b7504f4e27a Author: Bill Fischofer <bill.fischofer(a)linaro.org> Date: Tue Aug 30 20:37:25 2016 -0500 linux-generic: configure: add conditional shared library support Add support for the --enable-shared=[yes|no] configure options. --enable-shared=yes is the default and results in suppressing the use of inlines for ODP API functions to enhance ABI compatibility at potentially some performance cost. When --enable-shared=no is specified, inlines are allowed to boost performance at the possible loss of ABI compatibility. This patch addresses Bug https://bugs.linaro.org/show_bug.cgi?id=2490 Note: This patch is a port of the corresponding enhancements made to odp-dpdk with a few changes, mainly that inlines are now part of the odp/api/plat directory, and all ODP APIs are now treated uniformly with regard to whether inlining is permitted. Suggested-by: Zoltan Kiss <zoltan.kiss(a)linaro.org> Signed-off-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Balakrishna Garapati <balakrishna.garapati(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/configure.ac b/configure.ac index 6551287..982aff7 100644 --- a/configure.ac +++ b/configure.ac @@ -176,6 +176,13 @@ AM_CONDITIONAL([test_example], [test x$test_example = xyes ]) AM_CONDITIONAL([HAVE_DOXYGEN], [test "x${DOXYGEN}" = "xdoxygen"]) AM_CONDITIONAL([user_guide], [test "x${user_guides}" = "xyes" ]) AM_CONDITIONAL([HAVE_MSCGEN], [test "x${MSCGEN}" = "xmscgen"]) +if test x$enable_shared != xyes; +then + _ODP_INLINES="_ODP_INLINES" +else + _ODP_INLINES="_ODP_NO_INLINES" +fi +AC_SUBST(_ODP_INLINES) ########################################################################## # Setup doxygen documentation diff --git a/example/Makefile.inc b/example/Makefile.inc index 70ba2c0..9fb2f29 100644 --- a/example/Makefile.inc +++ b/example/Makefile.inc @@ -6,6 +6,7 @@ AM_CFLAGS += \ -I$(top_srcdir)/example \ -I$(top_srcdir)/platform/@with_platform@/include \ -I$(top_srcdir)/include/ \ - -I$(top_srcdir)/helper/include + -I$(top_srcdir)/helper/include \ + -I$(top_builddir)/platform/@with_platform@/include AM_LDFLAGS += -L$(LIB) diff --git a/helper/Makefile.am b/helper/Makefile.am index a82a11a..b6e6a1d 100644 --- a/helper/Makefile.am +++ b/helper/Makefile.am @@ -7,6 +7,7 @@ LIB = $(top_builddir)/lib AM_CFLAGS = -I$(srcdir)/include AM_CFLAGS += -I$(top_srcdir)/platform/@with_platform@/include AM_CFLAGS += -I$(top_srcdir)/include +AM_CFLAGS += -I$(top_builddir)/platform/@with_platform@/include AM_LDFLAGS += -version-number '$(ODPHELPER_LIBSO_VERSION)' diff --git a/platform/linux-generic/.gitignore b/platform/linux-generic/.gitignore new file mode 100644 index 0000000..ec6ca37 --- /dev/null +++ b/platform/linux-generic/.gitignore @@ -0,0 +1 @@ +include/odp/api/plat/inlines.h diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 0cfd0fe..ae3cbba 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -6,6 +6,7 @@ include $(top_srcdir)/platform/@with_platform@/Makefile.inc AM_CFLAGS += -I$(srcdir)/include AM_CFLAGS += -I$(top_srcdir)/include +AM_CFLAGS += -Iinclude include_HEADERS = \ $(top_srcdir)/include/odp.h \ @@ -60,15 +61,18 @@ odpapiinclude_HEADERS = \ odpapiplatincludedir= $(includedir)/odp/api/plat odpapiplatinclude_HEADERS = \ + $(srcdir)/include/odp/api/plat/atomic_inlines.h \ $(srcdir)/include/odp/api/plat/atomic_types.h \ $(srcdir)/include/odp/api/plat/barrier_types.h \ $(srcdir)/include/odp/api/plat/buffer_types.h \ + $(srcdir)/include/odp/api/plat/byteorder_inlines.h \ $(srcdir)/include/odp/api/plat/byteorder_types.h \ $(srcdir)/include/odp/api/plat/classification_types.h \ $(srcdir)/include/odp/api/plat/cpumask_types.h \ $(srcdir)/include/odp/api/plat/crypto_types.h \ $(srcdir)/include/odp/api/plat/event_types.h \ $(srcdir)/include/odp/api/plat/init_types.h \ + $(srcdir)/include/odp/api/plat/inlines.h \ $(srcdir)/include/odp/api/plat/packet_types.h \ $(srcdir)/include/odp/api/plat/packet_io_types.h \ $(srcdir)/include/odp/api/plat/pool_types.h \ @@ -79,7 +83,9 @@ odpapiplatinclude_HEADERS = \ $(srcdir)/include/odp/api/plat/shared_memory_types.h \ $(srcdir)/include/odp/api/plat/spinlock_types.h \ $(srcdir)/include/odp/api/plat/spinlock_recursive_types.h \ + $(srcdir)/include/odp/api/plat/std_clib_inlines.h \ $(srcdir)/include/odp/api/plat/strong_types.h \ + $(srcdir)/include/odp/api/plat/sync_inlines.h \ $(srcdir)/include/odp/api/plat/thread_types.h \ $(srcdir)/include/odp/api/plat/thrmask_types.h \ $(srcdir)/include/odp/api/plat/ticketlock_types.h \ @@ -134,6 +140,7 @@ __LIB__libodp_linux_la_SOURCES = \ odp_atomic.c \ odp_barrier.c \ odp_buffer.c \ + odp_byteorder.c \ odp_classification.c \ odp_cpu.c \ odp_cpumask.c \ @@ -173,6 +180,8 @@ __LIB__libodp_linux_la_SOURCES = \ odp_sorted_list.c \ odp_spinlock.c \ odp_spinlock_recursive.c \ + odp_std_clib.c \ + odp_sync.c \ odp_system_info.c \ odp_thread.c \ odp_thrmask.c \ diff --git a/platform/linux-generic/include/odp/api/atomic.h b/platform/linux-generic/include/odp/api/atomic.h index b487383..c18e68b 100644 --- a/platform/linux-generic/include/odp/api/atomic.h +++ b/platform/linux-generic/include/odp/api/atomic.h @@ -24,388 +24,10 @@ extern "C" { * @{ */ -static inline void odp_atomic_init_u32(odp_atomic_u32_t *atom, uint32_t val) -{ - __atomic_store_n(&atom->v, val, __ATOMIC_RELAXED); -} - -static inline uint32_t odp_atomic_load_u32(odp_atomic_u32_t *atom) -{ - return __atomic_load_n(&atom->v, __ATOMIC_RELAXED); -} - -static inline void odp_atomic_store_u32(odp_atomic_u32_t *atom, - uint32_t val) -{ - __atomic_store_n(&atom->v, val, __ATOMIC_RELAXED); -} - -static inline uint32_t odp_atomic_fetch_add_u32(odp_atomic_u32_t *atom, - uint32_t val) -{ - return __atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); -} - -static inline void odp_atomic_add_u32(odp_atomic_u32_t *atom, - uint32_t val) -{ - (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); -} - -static inline uint32_t odp_atomic_fetch_sub_u32(odp_atomic_u32_t *atom, - uint32_t val) -{ - return __atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); -} - -static inline void odp_atomic_sub_u32(odp_atomic_u32_t *atom, - uint32_t val) -{ - (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); -} - -static inline uint32_t odp_atomic_fetch_inc_u32(odp_atomic_u32_t *atom) -{ - return __atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); -} - -static inline void odp_atomic_inc_u32(odp_atomic_u32_t *atom) -{ - (void)__atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); -} - -static inline uint32_t odp_atomic_fetch_dec_u32(odp_atomic_u32_t *atom) -{ - return __atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); -} - -static inline void odp_atomic_dec_u32(odp_atomic_u32_t *atom) -{ - (void)__atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); -} - -static inline int odp_atomic_cas_u32(odp_atomic_u32_t *atom, uint32_t *old_val, - uint32_t new_val) -{ - return __atomic_compare_exchange_n(&atom->v, old_val, new_val, - 0 /* strong */, - __ATOMIC_RELAXED, - __ATOMIC_RELAXED); -} - -static inline uint32_t odp_atomic_xchg_u32(odp_atomic_u32_t *atom, - uint32_t new_val) -{ - return __atomic_exchange_n(&atom->v, new_val, __ATOMIC_RELAXED); -} - -static inline void odp_atomic_max_u32(odp_atomic_u32_t *atom, uint32_t new_max) -{ - uint32_t old_val; - - old_val = odp_atomic_load_u32(atom); - - while (new_max > old_val) { - if (odp_atomic_cas_u32(atom, &old_val, new_max)) - break; - } -} - -static inline void odp_atomic_min_u32(odp_atomic_u32_t *atom, uint32_t new_min) -{ - uint32_t old_val; - - old_val = odp_atomic_load_u32(atom); - - while (new_min < old_val) { - if (odp_atomic_cas_u32(atom, &old_val, new_min)) - break; - } -} - -static inline void odp_atomic_init_u64(odp_atomic_u64_t *atom, uint64_t val) -{ - atom->v = val; -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - __atomic_clear(&atom->lock, __ATOMIC_RELAXED); -#endif -} - -static inline uint64_t odp_atomic_load_u64(odp_atomic_u64_t *atom) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - return ATOMIC_OP(atom, (void)0); -#else - return __atomic_load_n(&atom->v, __ATOMIC_RELAXED); -#endif -} - -static inline void odp_atomic_store_u64(odp_atomic_u64_t *atom, - uint64_t val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v = val); -#else - __atomic_store_n(&atom->v, val, __ATOMIC_RELAXED); -#endif -} - -static inline uint64_t odp_atomic_fetch_add_u64(odp_atomic_u64_t *atom, - uint64_t val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - return ATOMIC_OP(atom, atom->v += val); -#else - return __atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); -#endif -} - -static inline void odp_atomic_add_u64(odp_atomic_u64_t *atom, uint64_t val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v += val); -#else - (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); -#endif -} - -static inline uint64_t odp_atomic_fetch_sub_u64(odp_atomic_u64_t *atom, - uint64_t val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - return ATOMIC_OP(atom, atom->v -= val); -#else - return __atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); -#endif -} - -static inline void odp_atomic_sub_u64(odp_atomic_u64_t *atom, uint64_t val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v -= val); -#else - (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); -#endif -} - -static inline uint64_t odp_atomic_fetch_inc_u64(odp_atomic_u64_t *atom) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - return ATOMIC_OP(atom, atom->v++); -#else - return __atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); -#endif -} - -static inline void odp_atomic_inc_u64(odp_atomic_u64_t *atom) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v++); -#else - (void)__atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); -#endif -} - -static inline uint64_t odp_atomic_fetch_dec_u64(odp_atomic_u64_t *atom) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - return ATOMIC_OP(atom, atom->v--); -#else - return __atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); +#include <odp/api/plat/inlines.h> +#ifdef _ODP_INLINES +#include <odp/api/plat/atomic_inlines.h> #endif -} - -static inline void odp_atomic_dec_u64(odp_atomic_u64_t *atom) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v--); -#else - (void)__atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); -#endif -} - -static inline int odp_atomic_cas_u64(odp_atomic_u64_t *atom, uint64_t *old_val, - uint64_t new_val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - int ret; - *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); - return ret; -#else - return __atomic_compare_exchange_n(&atom->v, old_val, new_val, - 0 /* strong */, - __ATOMIC_RELAXED, - __ATOMIC_RELAXED); -#endif -} - -static inline uint64_t odp_atomic_xchg_u64(odp_atomic_u64_t *atom, - uint64_t new_val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - return ATOMIC_OP(atom, atom->v = new_val); -#else - return __atomic_exchange_n(&atom->v, new_val, __ATOMIC_RELAXED); -#endif -} - -static inline void odp_atomic_max_u64(odp_atomic_u64_t *atom, uint64_t new_max) -{ - uint64_t old_val; - - old_val = odp_atomic_load_u64(atom); - - while (new_max > old_val) { - if (odp_atomic_cas_u64(atom, &old_val, new_max)) - break; - } -} - -static inline void odp_atomic_min_u64(odp_atomic_u64_t *atom, uint64_t new_min) -{ - uint64_t old_val; - - old_val = odp_atomic_load_u64(atom); - - while (new_min < old_val) { - if (odp_atomic_cas_u64(atom, &old_val, new_min)) - break; - } -} - -static inline uint32_t odp_atomic_load_acq_u32(odp_atomic_u32_t *atom) -{ - return __atomic_load_n(&atom->v, __ATOMIC_ACQUIRE); -} - -static inline void odp_atomic_store_rel_u32(odp_atomic_u32_t *atom, - uint32_t val) -{ - __atomic_store_n(&atom->v, val, __ATOMIC_RELEASE); -} - -static inline void odp_atomic_add_rel_u32(odp_atomic_u32_t *atom, - uint32_t val) -{ - (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELEASE); -} - -static inline void odp_atomic_sub_rel_u32(odp_atomic_u32_t *atom, - uint32_t val) -{ - (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELEASE); -} - -static inline int odp_atomic_cas_acq_u32(odp_atomic_u32_t *atom, - uint32_t *old_val, uint32_t new_val) -{ - return __atomic_compare_exchange_n(&atom->v, old_val, new_val, - 0 /* strong */, - __ATOMIC_ACQUIRE, - __ATOMIC_RELAXED); -} - -static inline int odp_atomic_cas_rel_u32(odp_atomic_u32_t *atom, - uint32_t *old_val, uint32_t new_val) -{ - return __atomic_compare_exchange_n(&atom->v, old_val, new_val, - 0 /* strong */, - __ATOMIC_RELEASE, - __ATOMIC_RELAXED); -} - -static inline int odp_atomic_cas_acq_rel_u32(odp_atomic_u32_t *atom, - uint32_t *old_val, - uint32_t new_val) -{ - return __atomic_compare_exchange_n(&atom->v, old_val, new_val, - 0 /* strong */, - __ATOMIC_ACQ_REL, - __ATOMIC_RELAXED); -} - -static inline uint64_t odp_atomic_load_acq_u64(odp_atomic_u64_t *atom) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - return ATOMIC_OP(atom, (void)0); -#else - return __atomic_load_n(&atom->v, __ATOMIC_ACQUIRE); -#endif -} - -static inline void odp_atomic_store_rel_u64(odp_atomic_u64_t *atom, - uint64_t val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v = val); -#else - __atomic_store_n(&atom->v, val, __ATOMIC_RELEASE); -#endif -} - -static inline void odp_atomic_add_rel_u64(odp_atomic_u64_t *atom, uint64_t val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v += val); -#else - (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELEASE); -#endif -} - -static inline void odp_atomic_sub_rel_u64(odp_atomic_u64_t *atom, uint64_t val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v -= val); -#else - (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELEASE); -#endif -} - -static inline int odp_atomic_cas_acq_u64(odp_atomic_u64_t *atom, - uint64_t *old_val, uint64_t new_val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - int ret; - *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); - return ret; -#else - return __atomic_compare_exchange_n(&atom->v, old_val, new_val, - 0 /* strong */, - __ATOMIC_ACQUIRE, - __ATOMIC_RELAXED); -#endif -} - -static inline int odp_atomic_cas_rel_u64(odp_atomic_u64_t *atom, - uint64_t *old_val, uint64_t new_val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - int ret; - *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); - return ret; -#else - return __atomic_compare_exchange_n(&atom->v, old_val, new_val, - 0 /* strong */, - __ATOMIC_RELEASE, - __ATOMIC_RELAXED); -#endif -} - -static inline int odp_atomic_cas_acq_rel_u64(odp_atomic_u64_t *atom, - uint64_t *old_val, - uint64_t new_val) -{ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - int ret; - *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); - return ret; -#else - return __atomic_compare_exchange_n(&atom->v, old_val, new_val, - 0 /* strong */, - __ATOMIC_ACQ_REL, - __ATOMIC_RELAXED); -#endif -} /** * @} diff --git a/platform/linux-generic/include/odp/api/byteorder.h b/platform/linux-generic/include/odp/api/byteorder.h index c347be0..84d1173 100644 --- a/platform/linux-generic/include/odp/api/byteorder.h +++ b/platform/linux-generic/include/odp/api/byteorder.h @@ -17,124 +17,18 @@ extern "C" { #endif -#include <odp/api/plat/byteorder_types.h> #include <odp/api/std_types.h> +#include <odp/api/plat/byteorder_types.h> #include <odp/api/compiler.h> /** @ingroup odp_compiler_optim * @{ */ -static inline uint16_t odp_be_to_cpu_16(odp_u16be_t be16) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return __odp_builtin_bswap16((__odp_force uint16_t)be16); -#else - return (__odp_force uint16_t)be16; -#endif -} - -static inline uint32_t odp_be_to_cpu_32(odp_u32be_t be32) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return __builtin_bswap32((__odp_force uint32_t)be32); -#else - return (__odp_force uint32_t)be32; -#endif -} - -static inline uint64_t odp_be_to_cpu_64(odp_u64be_t be64) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return __builtin_bswap64((__odp_force uint64_t)be64); -#else - return (__odp_force uint64_t)be64; -#endif -} - - -static inline odp_u16be_t odp_cpu_to_be_16(uint16_t cpu16) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return (__odp_force odp_u16be_t)__odp_builtin_bswap16(cpu16); -#else - return (__odp_force odp_u16be_t)cpu16; -#endif -} - -static inline odp_u32be_t odp_cpu_to_be_32(uint32_t cpu32) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return (__odp_force odp_u32be_t)__builtin_bswap32(cpu32); -#else - return (__odp_force odp_u32be_t)cpu32; +#include <odp/api/plat/inlines.h> +#ifdef _ODP_INLINES +#include <odp/api/plat/byteorder_inlines.h> #endif -} - -static inline odp_u64be_t odp_cpu_to_be_64(uint64_t cpu64) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return (__odp_force odp_u64be_t)__builtin_bswap64(cpu64); -#else - return (__odp_force odp_u64be_t)cpu64; -#endif -} - - -static inline uint16_t odp_le_to_cpu_16(odp_u16le_t le16) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return (__odp_force uint16_t)le16; -#else - return __odp_builtin_bswap16((__odp_force uint16_t)le16); -#endif -} - -static inline uint32_t odp_le_to_cpu_32(odp_u32le_t le32) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return (__odp_force uint32_t)le32; -#else - return __builtin_bswap32((__odp_force uint32_t)le32); -#endif -} - -static inline uint64_t odp_le_to_cpu_64(odp_u64le_t le64) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return (__odp_force uint64_t)le64; -#else - return __builtin_bswap64((__odp_force uint64_t)le64); -#endif -} - - -static inline odp_u16le_t odp_cpu_to_le_16(uint16_t cpu16) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return (__odp_force odp_u16le_t)cpu16; -#else - return (__odp_force odp_u16le_t)__odp_builtin_bswap16(cpu16); -#endif -} - -static inline odp_u32le_t odp_cpu_to_le_32(uint32_t cpu32) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return (__odp_force odp_u32le_t)cpu32; -#else - return (__odp_force odp_u32le_t)__builtin_bswap32(cpu32); -#endif -} - -static inline odp_u64le_t odp_cpu_to_le_64(uint64_t cpu64) -{ -#if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN - return (__odp_force odp_u64le_t)cpu64; -#else - return (__odp_force odp_u64le_t)__builtin_bswap64(cpu64); -#endif -} /** * @} diff --git a/platform/linux-generic/include/odp/api/atomic.h b/platform/linux-generic/include/odp/api/plat/atomic_inlines.h similarity index 58% copy from platform/linux-generic/include/odp/api/atomic.h copy to platform/linux-generic/include/odp/api/plat/atomic_inlines.h index b487383..4471f2e 100644 --- a/platform/linux-generic/include/odp/api/atomic.h +++ b/platform/linux-generic/include/odp/api/plat/atomic_inlines.h @@ -1,91 +1,75 @@ -/* Copyright (c) 2013, Linaro Limited +/* Copyright (c) 2016, Linaro Limited * All rights reserved. * - * SPDX-License-Identifier: BSD-3-Clause + * SPDX-License-Identifier: BSD-3-Clause */ /** * @file * - * ODP atomic operations + * ODP Atomic inline functions */ -#ifndef ODP_PLAT_ATOMIC_H_ -#define ODP_PLAT_ATOMIC_H_ +#ifndef _ODP_PLAT_ATOMIC_INLINES_H_ +#define _ODP_PLAT_ATOMIC_INLINES_H_ -#ifdef __cplusplus -extern "C" { -#endif - -#include <odp/api/align.h> -#include <odp/api/plat/atomic_types.h> - -/** @ingroup odp_atomic - * @{ - */ - -static inline void odp_atomic_init_u32(odp_atomic_u32_t *atom, uint32_t val) +_STATIC void odp_atomic_init_u32(odp_atomic_u32_t *atom, uint32_t val) { __atomic_store_n(&atom->v, val, __ATOMIC_RELAXED); } -static inline uint32_t odp_atomic_load_u32(odp_atomic_u32_t *atom) +_STATIC uint32_t odp_atomic_load_u32(odp_atomic_u32_t *atom) { return __atomic_load_n(&atom->v, __ATOMIC_RELAXED); } -static inline void odp_atomic_store_u32(odp_atomic_u32_t *atom, - uint32_t val) +_STATIC void odp_atomic_store_u32(odp_atomic_u32_t *atom, uint32_t val) { __atomic_store_n(&atom->v, val, __ATOMIC_RELAXED); } -static inline uint32_t odp_atomic_fetch_add_u32(odp_atomic_u32_t *atom, - uint32_t val) +_STATIC uint32_t odp_atomic_fetch_add_u32(odp_atomic_u32_t *atom, uint32_t val) { return __atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); } -static inline void odp_atomic_add_u32(odp_atomic_u32_t *atom, - uint32_t val) +_STATIC void odp_atomic_add_u32(odp_atomic_u32_t *atom, uint32_t val) { (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); } -static inline uint32_t odp_atomic_fetch_sub_u32(odp_atomic_u32_t *atom, - uint32_t val) +_STATIC uint32_t odp_atomic_fetch_sub_u32(odp_atomic_u32_t *atom, uint32_t val) { return __atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); } -static inline void odp_atomic_sub_u32(odp_atomic_u32_t *atom, - uint32_t val) +_STATIC void odp_atomic_sub_u32(odp_atomic_u32_t *atom, uint32_t val) { (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); } -static inline uint32_t odp_atomic_fetch_inc_u32(odp_atomic_u32_t *atom) +_STATIC uint32_t odp_atomic_fetch_inc_u32(odp_atomic_u32_t *atom) { return __atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); } -static inline void odp_atomic_inc_u32(odp_atomic_u32_t *atom) +_STATIC void odp_atomic_inc_u32(odp_atomic_u32_t *atom) { (void)__atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); } -static inline uint32_t odp_atomic_fetch_dec_u32(odp_atomic_u32_t *atom) +_STATIC uint32_t odp_atomic_fetch_dec_u32(odp_atomic_u32_t *atom) { return __atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); } -static inline void odp_atomic_dec_u32(odp_atomic_u32_t *atom) +_STATIC void odp_atomic_dec_u32(odp_atomic_u32_t *atom) { (void)__atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); } -static inline int odp_atomic_cas_u32(odp_atomic_u32_t *atom, uint32_t *old_val, - uint32_t new_val) +_STATIC int odp_atomic_cas_u32(odp_atomic_u32_t *atom, uint32_t *old_val, + uint32_t new_val) { return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, @@ -93,13 +77,12 @@ static inline int odp_atomic_cas_u32(odp_atomic_u32_t *atom, uint32_t *old_val, __ATOMIC_RELAXED); } -static inline uint32_t odp_atomic_xchg_u32(odp_atomic_u32_t *atom, - uint32_t new_val) +_STATIC uint32_t odp_atomic_xchg_u32(odp_atomic_u32_t *atom, uint32_t new_val) { return __atomic_exchange_n(&atom->v, new_val, __ATOMIC_RELAXED); } -static inline void odp_atomic_max_u32(odp_atomic_u32_t *atom, uint32_t new_max) +_STATIC void odp_atomic_max_u32(odp_atomic_u32_t *atom, uint32_t new_max) { uint32_t old_val; @@ -111,7 +94,7 @@ static inline void odp_atomic_max_u32(odp_atomic_u32_t *atom, uint32_t new_max) } } -static inline void odp_atomic_min_u32(odp_atomic_u32_t *atom, uint32_t new_min) +_STATIC void odp_atomic_min_u32(odp_atomic_u32_t *atom, uint32_t new_min) { uint32_t old_val; @@ -123,7 +106,7 @@ static inline void odp_atomic_min_u32(odp_atomic_u32_t *atom, uint32_t new_min) } } -static inline void odp_atomic_init_u64(odp_atomic_u64_t *atom, uint64_t val) +_STATIC void odp_atomic_init_u64(odp_atomic_u64_t *atom, uint64_t val) { atom->v = val; #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 @@ -131,7 +114,7 @@ static inline void odp_atomic_init_u64(odp_atomic_u64_t *atom, uint64_t val) #endif } -static inline uint64_t odp_atomic_load_u64(odp_atomic_u64_t *atom) +_STATIC uint64_t odp_atomic_load_u64(odp_atomic_u64_t *atom) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, (void)0); @@ -140,8 +123,7 @@ static inline uint64_t odp_atomic_load_u64(odp_atomic_u64_t *atom) #endif } -static inline void odp_atomic_store_u64(odp_atomic_u64_t *atom, - uint64_t val) +_STATIC void odp_atomic_store_u64(odp_atomic_u64_t *atom, uint64_t val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v = val); @@ -150,8 +132,7 @@ static inline void odp_atomic_store_u64(odp_atomic_u64_t *atom, #endif } -static inline uint64_t odp_atomic_fetch_add_u64(odp_atomic_u64_t *atom, - uint64_t val) +_STATIC uint64_t odp_atomic_fetch_add_u64(odp_atomic_u64_t *atom, uint64_t val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v += val); @@ -160,7 +141,7 @@ static inline uint64_t odp_atomic_fetch_add_u64(odp_atomic_u64_t *atom, #endif } -static inline void odp_atomic_add_u64(odp_atomic_u64_t *atom, uint64_t val) +_STATIC void odp_atomic_add_u64(odp_atomic_u64_t *atom, uint64_t val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v += val); @@ -169,8 +150,7 @@ static inline void odp_atomic_add_u64(odp_atomic_u64_t *atom, uint64_t val) #endif } -static inline uint64_t odp_atomic_fetch_sub_u64(odp_atomic_u64_t *atom, - uint64_t val) +_STATIC uint64_t odp_atomic_fetch_sub_u64(odp_atomic_u64_t *atom, uint64_t val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v -= val); @@ -179,7 +159,7 @@ static inline uint64_t odp_atomic_fetch_sub_u64(odp_atomic_u64_t *atom, #endif } -static inline void odp_atomic_sub_u64(odp_atomic_u64_t *atom, uint64_t val) +_STATIC void odp_atomic_sub_u64(odp_atomic_u64_t *atom, uint64_t val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v -= val); @@ -188,7 +168,7 @@ static inline void odp_atomic_sub_u64(odp_atomic_u64_t *atom, uint64_t val) #endif } -static inline uint64_t odp_atomic_fetch_inc_u64(odp_atomic_u64_t *atom) +_STATIC uint64_t odp_atomic_fetch_inc_u64(odp_atomic_u64_t *atom) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v++); @@ -197,7 +177,7 @@ static inline uint64_t odp_atomic_fetch_inc_u64(odp_atomic_u64_t *atom) #endif } -static inline void odp_atomic_inc_u64(odp_atomic_u64_t *atom) +_STATIC void odp_atomic_inc_u64(odp_atomic_u64_t *atom) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v++); @@ -206,7 +186,7 @@ static inline void odp_atomic_inc_u64(odp_atomic_u64_t *atom) #endif } -static inline uint64_t odp_atomic_fetch_dec_u64(odp_atomic_u64_t *atom) +_STATIC uint64_t odp_atomic_fetch_dec_u64(odp_atomic_u64_t *atom) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v--); @@ -215,7 +195,7 @@ static inline uint64_t odp_atomic_fetch_dec_u64(odp_atomic_u64_t *atom) #endif } -static inline void odp_atomic_dec_u64(odp_atomic_u64_t *atom) +_STATIC void odp_atomic_dec_u64(odp_atomic_u64_t *atom) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v--); @@ -224,8 +204,8 @@ static inline void odp_atomic_dec_u64(odp_atomic_u64_t *atom) #endif } -static inline int odp_atomic_cas_u64(odp_atomic_u64_t *atom, uint64_t *old_val, - uint64_t new_val) +_STATIC int odp_atomic_cas_u64(odp_atomic_u64_t *atom, uint64_t *old_val, + uint64_t new_val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 int ret; @@ -239,8 +219,7 @@ static inline int odp_atomic_cas_u64(odp_atomic_u64_t *atom, uint64_t *old_val, #endif } -static inline uint64_t odp_atomic_xchg_u64(odp_atomic_u64_t *atom, - uint64_t new_val) +_STATIC uint64_t odp_atomic_xchg_u64(odp_atomic_u64_t *atom, uint64_t new_val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v = new_val); @@ -249,7 +228,7 @@ static inline uint64_t odp_atomic_xchg_u64(odp_atomic_u64_t *atom, #endif } -static inline void odp_atomic_max_u64(odp_atomic_u64_t *atom, uint64_t new_max) +_STATIC void odp_atomic_max_u64(odp_atomic_u64_t *atom, uint64_t new_max) { uint64_t old_val; @@ -261,7 +240,7 @@ static inline void odp_atomic_max_u64(odp_atomic_u64_t *atom, uint64_t new_max) } } -static inline void odp_atomic_min_u64(odp_atomic_u64_t *atom, uint64_t new_min) +_STATIC void odp_atomic_min_u64(odp_atomic_u64_t *atom, uint64_t new_min) { uint64_t old_val; @@ -273,31 +252,28 @@ static inline void odp_atomic_min_u64(odp_atomic_u64_t *atom, uint64_t new_min) } } -static inline uint32_t odp_atomic_load_acq_u32(odp_atomic_u32_t *atom) +_STATIC uint32_t odp_atomic_load_acq_u32(odp_atomic_u32_t *atom) { return __atomic_load_n(&atom->v, __ATOMIC_ACQUIRE); } -static inline void odp_atomic_store_rel_u32(odp_atomic_u32_t *atom, - uint32_t val) +_STATIC void odp_atomic_store_rel_u32(odp_atomic_u32_t *atom, uint32_t val) { __atomic_store_n(&atom->v, val, __ATOMIC_RELEASE); } -static inline void odp_atomic_add_rel_u32(odp_atomic_u32_t *atom, - uint32_t val) +_STATIC void odp_atomic_add_rel_u32(odp_atomic_u32_t *atom, uint32_t val) { (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELEASE); } -static inline void odp_atomic_sub_rel_u32(odp_atomic_u32_t *atom, - uint32_t val) +_STATIC void odp_atomic_sub_rel_u32(odp_atomic_u32_t *atom, uint32_t val) { (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELEASE); } -static inline int odp_atomic_cas_acq_u32(odp_atomic_u32_t *atom, - uint32_t *old_val, uint32_t new_val) +_STATIC int odp_atomic_cas_acq_u32(odp_atomic_u32_t *atom, + uint32_t *old_val, uint32_t new_val) { return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, @@ -305,8 +281,8 @@ static inline int odp_atomic_cas_acq_u32(odp_atomic_u32_t *atom, __ATOMIC_RELAXED); } -static inline int odp_atomic_cas_rel_u32(odp_atomic_u32_t *atom, - uint32_t *old_val, uint32_t new_val) +_STATIC int odp_atomic_cas_rel_u32(odp_atomic_u32_t *atom, + uint32_t *old_val, uint32_t new_val) { return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, @@ -314,9 +290,9 @@ static inline int odp_atomic_cas_rel_u32(odp_atomic_u32_t *atom, __ATOMIC_RELAXED); } -static inline int odp_atomic_cas_acq_rel_u32(odp_atomic_u32_t *atom, - uint32_t *old_val, - uint32_t new_val) +_STATIC int odp_atomic_cas_acq_rel_u32(odp_atomic_u32_t *atom, + uint32_t *old_val, + uint32_t new_val) { return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, @@ -324,7 +300,7 @@ static inline int odp_atomic_cas_acq_rel_u32(odp_atomic_u32_t *atom, __ATOMIC_RELAXED); } -static inline uint64_t odp_atomic_load_acq_u64(odp_atomic_u64_t *atom) +_STATIC uint64_t odp_atomic_load_acq_u64(odp_atomic_u64_t *atom) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, (void)0); @@ -333,8 +309,7 @@ static inline uint64_t odp_atomic_load_acq_u64(odp_atomic_u64_t *atom) #endif } -static inline void odp_atomic_store_rel_u64(odp_atomic_u64_t *atom, - uint64_t val) +_STATIC void odp_atomic_store_rel_u64(odp_atomic_u64_t *atom, uint64_t val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v = val); @@ -343,7 +318,7 @@ static inline void odp_atomic_store_rel_u64(odp_atomic_u64_t *atom, #endif } -static inline void odp_atomic_add_rel_u64(odp_atomic_u64_t *atom, uint64_t val) +_STATIC void odp_atomic_add_rel_u64(odp_atomic_u64_t *atom, uint64_t val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v += val); @@ -352,7 +327,7 @@ static inline void odp_atomic_add_rel_u64(odp_atomic_u64_t *atom, uint64_t val) #endif } -static inline void odp_atomic_sub_rel_u64(odp_atomic_u64_t *atom, uint64_t val) +_STATIC void odp_atomic_sub_rel_u64(odp_atomic_u64_t *atom, uint64_t val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v -= val); @@ -361,8 +336,8 @@ static inline void odp_atomic_sub_rel_u64(odp_atomic_u64_t *atom, uint64_t val) #endif } -static inline int odp_atomic_cas_acq_u64(odp_atomic_u64_t *atom, - uint64_t *old_val, uint64_t new_val) +_STATIC int odp_atomic_cas_acq_u64(odp_atomic_u64_t *atom, + uint64_t *old_val, uint64_t new_val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 int ret; @@ -376,8 +351,8 @@ static inline int odp_atomic_cas_acq_u64(odp_atomic_u64_t *atom, #endif } -static inline int odp_atomic_cas_rel_u64(odp_atomic_u64_t *atom, - uint64_t *old_val, uint64_t new_val) +_STATIC int odp_atomic_cas_rel_u64(odp_atomic_u64_t *atom, + uint64_t *old_val, uint64_t new_val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 int ret; @@ -391,9 +366,9 @@ static inline int odp_atomic_cas_rel_u64(odp_atomic_u64_t *atom, #endif } -static inline int odp_atomic_cas_acq_rel_u64(odp_atomic_u64_t *atom, - uint64_t *old_val, - uint64_t new_val) +_STATIC int odp_atomic_cas_acq_rel_u64(odp_atomic_u64_t *atom, + uint64_t *old_val, + uint64_t new_val) { #if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 int ret; @@ -407,14 +382,4 @@ static inline int odp_atomic_cas_acq_rel_u64(odp_atomic_u64_t *atom, #endif } -/** - * @} - */ - -#include <odp/api/spec/atomic.h> - -#ifdef __cplusplus -} -#endif - #endif diff --git a/platform/linux-generic/include/odp/api/byteorder.h b/platform/linux-generic/include/odp/api/plat/byteorder_inlines.h similarity index 69% copy from platform/linux-generic/include/odp/api/byteorder.h copy to platform/linux-generic/include/odp/api/plat/byteorder_inlines.h index c347be0..c91a3b1 100644 --- a/platform/linux-generic/include/odp/api/byteorder.h +++ b/platform/linux-generic/include/odp/api/plat/byteorder_inlines.h @@ -1,4 +1,4 @@ -/* Copyright (c) 2014, Linaro Limited +/* Copyright (c) 2016, Linaro Limited * All rights reserved. * * SPDX-License-Identifier: BSD-3-Clause @@ -10,22 +10,18 @@ * ODP byteorder */ -#ifndef ODP_PLAT_BYTEORDER_H_ -#define ODP_PLAT_BYTEORDER_H_ +#ifndef ODP_PLAT_BYTEORDER_INLINES_H_ +#define ODP_PLAT_BYTEORDER_INLINES_H_ #ifdef __cplusplus extern "C" { #endif -#include <odp/api/plat/byteorder_types.h> -#include <odp/api/std_types.h> -#include <odp/api/compiler.h> - /** @ingroup odp_compiler_optim * @{ */ -static inline uint16_t odp_be_to_cpu_16(odp_u16be_t be16) +_STATIC uint16_t odp_be_to_cpu_16(odp_u16be_t be16) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return __odp_builtin_bswap16((__odp_force uint16_t)be16); @@ -34,7 +30,7 @@ static inline uint16_t odp_be_to_cpu_16(odp_u16be_t be16) #endif } -static inline uint32_t odp_be_to_cpu_32(odp_u32be_t be32) +_STATIC uint32_t odp_be_to_cpu_32(odp_u32be_t be32) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return __builtin_bswap32((__odp_force uint32_t)be32); @@ -43,7 +39,7 @@ static inline uint32_t odp_be_to_cpu_32(odp_u32be_t be32) #endif } -static inline uint64_t odp_be_to_cpu_64(odp_u64be_t be64) +_STATIC uint64_t odp_be_to_cpu_64(odp_u64be_t be64) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return __builtin_bswap64((__odp_force uint64_t)be64); @@ -52,8 +48,7 @@ static inline uint64_t odp_be_to_cpu_64(odp_u64be_t be64) #endif } - -static inline odp_u16be_t odp_cpu_to_be_16(uint16_t cpu16) +_STATIC odp_u16be_t odp_cpu_to_be_16(uint16_t cpu16) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return (__odp_force odp_u16be_t)__odp_builtin_bswap16(cpu16); @@ -62,7 +57,7 @@ static inline odp_u16be_t odp_cpu_to_be_16(uint16_t cpu16) #endif } -static inline odp_u32be_t odp_cpu_to_be_32(uint32_t cpu32) +_STATIC odp_u32be_t odp_cpu_to_be_32(uint32_t cpu32) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return (__odp_force odp_u32be_t)__builtin_bswap32(cpu32); @@ -71,7 +66,7 @@ static inline odp_u32be_t odp_cpu_to_be_32(uint32_t cpu32) #endif } -static inline odp_u64be_t odp_cpu_to_be_64(uint64_t cpu64) +_STATIC odp_u64be_t odp_cpu_to_be_64(uint64_t cpu64) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return (__odp_force odp_u64be_t)__builtin_bswap64(cpu64); @@ -80,8 +75,7 @@ static inline odp_u64be_t odp_cpu_to_be_64(uint64_t cpu64) #endif } - -static inline uint16_t odp_le_to_cpu_16(odp_u16le_t le16) +_STATIC uint16_t odp_le_to_cpu_16(odp_u16le_t le16) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return (__odp_force uint16_t)le16; @@ -90,7 +84,7 @@ static inline uint16_t odp_le_to_cpu_16(odp_u16le_t le16) #endif } -static inline uint32_t odp_le_to_cpu_32(odp_u32le_t le32) +_STATIC uint32_t odp_le_to_cpu_32(odp_u32le_t le32) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return (__odp_force uint32_t)le32; @@ -99,7 +93,7 @@ static inline uint32_t odp_le_to_cpu_32(odp_u32le_t le32) #endif } -static inline uint64_t odp_le_to_cpu_64(odp_u64le_t le64) +_STATIC uint64_t odp_le_to_cpu_64(odp_u64le_t le64) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return (__odp_force uint64_t)le64; @@ -108,8 +102,7 @@ static inline uint64_t odp_le_to_cpu_64(odp_u64le_t le64) #endif } - -static inline odp_u16le_t odp_cpu_to_le_16(uint16_t cpu16) +_STATIC odp_u16le_t odp_cpu_to_le_16(uint16_t cpu16) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return (__odp_force odp_u16le_t)cpu16; @@ -118,7 +111,7 @@ static inline odp_u16le_t odp_cpu_to_le_16(uint16_t cpu16) #endif } -static inline odp_u32le_t odp_cpu_to_le_32(uint32_t cpu32) +_STATIC odp_u32le_t odp_cpu_to_le_32(uint32_t cpu32) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return (__odp_force odp_u32le_t)cpu32; @@ -127,7 +120,7 @@ static inline odp_u32le_t odp_cpu_to_le_32(uint32_t cpu32) #endif } -static inline odp_u64le_t odp_cpu_to_le_64(uint64_t cpu64) +_STATIC odp_u64le_t odp_cpu_to_le_64(uint64_t cpu64) { #if ODP_BYTE_ORDER == ODP_LITTLE_ENDIAN return (__odp_force odp_u64le_t)cpu64; @@ -140,8 +133,6 @@ static inline odp_u64le_t odp_cpu_to_le_64(uint64_t cpu64) * @} */ -#include <odp/api/spec/byteorder.h> - #ifdef __cplusplus } #endif diff --git a/platform/linux-generic/include/odp/api/plat/inlines.h.in b/platform/linux-generic/include/odp/api/plat/inlines.h.in new file mode 100644 index 0000000..5d8c0dc --- /dev/null +++ b/platform/linux-generic/include/odp/api/plat/inlines.h.in @@ -0,0 +1,33 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODP platform inline functions + */ + +#ifndef ODP_PLAT_INLINES_H_ +#define ODP_PLAT_INLINES_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#define @_ODP_INLINES@ + +#ifdef _ODP_INLINES +#define _STATIC static inline +#else +#define _STATIC +#endif + +#ifdef __cplusplus +} +#endif + + +#endif /* ODP_PLAT_INLINES_H_ */ diff --git a/platform/linux-generic/include/odp/api/std_clib.h b/platform/linux-generic/include/odp/api/plat/std_clib_inlines.h similarity index 52% copy from platform/linux-generic/include/odp/api/std_clib.h copy to platform/linux-generic/include/odp/api/plat/std_clib_inlines.h index 40c0ea8..2e833e4 100644 --- a/platform/linux-generic/include/odp/api/std_clib.h +++ b/platform/linux-generic/include/odp/api/plat/std_clib_inlines.h @@ -1,11 +1,11 @@ -/* Copyright (c) 2015, Linaro Limited +/* Copyright (c) 2016, Linaro Limited * All rights reserved. * * SPDX-License-Identifier: BSD-3-Clause */ -#ifndef ODP_PLAT_STD_CLIB_H_ -#define ODP_PLAT_STD_CLIB_H_ +#ifndef ODP_PLAT_STD_CLIB_INLINE_H_ +#define ODP_PLAT_STD_CLIB_INLINE_H_ #ifdef __cplusplus extern "C" { @@ -14,17 +14,17 @@ extern "C" { #include <odp/api/spec/std_types.h> #include <string.h> -static inline void *odp_memcpy(void *dst, const void *src, size_t num) +_STATIC void *odp_memcpy(void *dst, const void *src, size_t num) { return memcpy(dst, src, num); } -static inline void *odp_memset(void *ptr, int value, size_t num) +_STATIC void *odp_memset(void *ptr, int value, size_t num) { return memset(ptr, value, num); } -static inline int odp_memcmp(const void *ptr1, const void *ptr2, size_t num) +_STATIC int odp_memcmp(const void *ptr1, const void *ptr2, size_t num) { return memcmp(ptr1, ptr2, num); } diff --git a/platform/linux-generic/include/odp/api/sync.h b/platform/linux-generic/include/odp/api/plat/sync_inlines.h similarity index 58% copy from platform/linux-generic/include/odp/api/sync.h copy to platform/linux-generic/include/odp/api/plat/sync_inlines.h index 45fe6bd..245c750 100644 --- a/platform/linux-generic/include/odp/api/sync.h +++ b/platform/linux-generic/include/odp/api/plat/sync_inlines.h @@ -1,4 +1,4 @@ -/* Copyright (c) 2013, Linaro Limited +/* Copyright (c) 2016, Linaro Limited * All rights reserved. * * SPDX-License-Identifier: BSD-3-Clause @@ -7,11 +7,11 @@ /** * @file * - * ODP synchronisation + * ODP synchronisation inlines */ -#ifndef ODP_PLAT_SYNC_H_ -#define ODP_PLAT_SYNC_H_ +#ifndef ODP_PLAT_SYNC_INLINE_H_ +#define ODP_PLAT_SYNC_INLINE_H_ #ifdef __cplusplus extern "C" { @@ -21,17 +21,17 @@ extern "C" { * @{ */ -static inline void odp_mb_release(void) +_STATIC void odp_mb_release(void) { __atomic_thread_fence(__ATOMIC_RELEASE); } -static inline void odp_mb_acquire(void) +_STATIC void odp_mb_acquire(void) { __atomic_thread_fence(__ATOMIC_ACQUIRE); } -static inline void odp_mb_full(void) +_STATIC void odp_mb_full(void) { __atomic_thread_fence(__ATOMIC_SEQ_CST); } @@ -40,8 +40,6 @@ static inline void odp_mb_full(void) * @} */ -#include <odp/api/spec/sync.h> - #ifdef __cplusplus } #endif diff --git a/platform/linux-generic/include/odp/api/std_clib.h b/platform/linux-generic/include/odp/api/std_clib.h index 40c0ea8..c498f68 100644 --- a/platform/linux-generic/include/odp/api/std_clib.h +++ b/platform/linux-generic/include/odp/api/std_clib.h @@ -14,20 +14,12 @@ extern "C" { #include <odp/api/spec/std_types.h> #include <string.h> -static inline void *odp_memcpy(void *dst, const void *src, size_t num) -{ - return memcpy(dst, src, num); -} - -static inline void *odp_memset(void *ptr, int value, size_t num) -{ - return memset(ptr, value, num); -} +#include <odp/api/plat/inlines.h> +#ifdef _ODP_INLINES +#include <odp/api/plat/std_clib_inlines.h> +#endif -static inline int odp_memcmp(const void *ptr1, const void *ptr2, size_t num) -{ - return memcmp(ptr1, ptr2, num); -} +#include <odp/api/spec/std_clib.h> #ifdef __cplusplus } diff --git a/platform/linux-generic/include/odp/api/sync.h b/platform/linux-generic/include/odp/api/sync.h index 45fe6bd..d2becb9 100644 --- a/platform/linux-generic/include/odp/api/sync.h +++ b/platform/linux-generic/include/odp/api/sync.h @@ -21,20 +21,10 @@ extern "C" { * @{ */ -static inline void odp_mb_release(void) -{ - __atomic_thread_fence(__ATOMIC_RELEASE); -} - -static inline void odp_mb_acquire(void) -{ - __atomic_thread_fence(__ATOMIC_ACQUIRE); -} - -static inline void odp_mb_full(void) -{ - __atomic_thread_fence(__ATOMIC_SEQ_CST); -} +#include <odp/api/plat/inlines.h> +#ifdef _ODP_INLINES +#include <odp/api/plat/sync_inlines.h> +#endif /** * @} diff --git a/platform/linux-generic/m4/configure.m4 b/platform/linux-generic/m4/configure.m4 index 1b1b883..6fb05c0 100644 --- a/platform/linux-generic/m4/configure.m4 +++ b/platform/linux-generic/m4/configure.m4 @@ -36,4 +36,5 @@ m4_include([platform/linux-generic/m4/odp_dpdk.m4]) m4_include([platform/linux-generic/m4/odp_ipc.m4]) m4_include([platform/linux-generic/m4/odp_schedule.m4]) -AC_CONFIG_FILES([platform/linux-generic/Makefile]) +AC_CONFIG_FILES([platform/linux-generic/Makefile + platform/linux-generic/include/odp/api/plat/inlines.h]) diff --git a/platform/linux-generic/odp_atomic.c b/platform/linux-generic/odp_atomic.c index 680e66d..e9a3ed0 100644 --- a/platform/linux-generic/odp_atomic.c +++ b/platform/linux-generic/odp_atomic.c @@ -5,6 +5,9 @@ */ #include <odp/api/atomic.h> +#ifndef _ODP_INLINES +#include <odp/api/plat/atomic_inlines.h> +#endif int odp_atomic_lock_free_u64(odp_atomic_op_t *atomic_op) { diff --git a/platform/linux-generic/odp_byteorder.c b/platform/linux-generic/odp_byteorder.c new file mode 100644 index 0000000..fc87291 --- /dev/null +++ b/platform/linux-generic/odp_byteorder.c @@ -0,0 +1,10 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp/api/byteorder.h> +#ifndef _ODP_INLINES +#include <odp/api/plat/byteorder_inlines.h> +#endif diff --git a/platform/linux-generic/odp_std_clib.c b/platform/linux-generic/odp_std_clib.c new file mode 100644 index 0000000..611ba12 --- /dev/null +++ b/platform/linux-generic/odp_std_clib.c @@ -0,0 +1,10 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp/api/std_clib.h> +#ifndef _ODP_INLINES +#include <odp/api/plat/std_clib_inlines.h> +#endif diff --git a/platform/linux-generic/odp_sync.c b/platform/linux-generic/odp_sync.c new file mode 100644 index 0000000..f31c389 --- /dev/null +++ b/platform/linux-generic/odp_sync.c @@ -0,0 +1,10 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp/api/sync.h> +#ifndef _ODP_INLINES +#include <odp/api/plat/sync_inlines.h> +#endif ----------------------------------------------------------------------- Summary of changes: configure.ac | 7 + example/Makefile.inc | 3 +- helper/Makefile.am | 1 + platform/linux-generic/.gitignore | 1 + platform/linux-generic/Makefile.am | 9 + platform/linux-generic/include/odp/api/atomic.h | 384 +-------------------- platform/linux-generic/include/odp/api/byteorder.h | 114 +----- .../odp/api/{atomic.h => plat/atomic_inlines.h} | 153 ++++---- .../api/{byteorder.h => plat/byteorder_inlines.h} | 39 +-- .../include/odp/api/plat/inlines.h.in | 33 ++ .../api/{std_clib.h => plat/std_clib_inlines.h} | 12 +- .../odp/api/{sync.h => plat/sync_inlines.h} | 16 +- platform/linux-generic/include/odp/api/std_clib.h | 18 +- platform/linux-generic/include/odp/api/sync.h | 18 +- platform/linux-generic/m4/configure.m4 | 3 +- platform/linux-generic/odp_atomic.c | 3 + .../linux-generic/odp_byteorder.c | 10 +- .../linux-generic/odp_std_clib.c | 10 +- .../linux-generic/odp_sync.c | 5 +- 19 files changed, 173 insertions(+), 666 deletions(-) create mode 100644 platform/linux-generic/.gitignore copy platform/linux-generic/include/odp/api/{atomic.h => plat/atomic_inlines.h} (58%) copy platform/linux-generic/include/odp/api/{byteorder.h => plat/byteorder_inlines.h} (69%) create mode 100644 platform/linux-generic/include/odp/api/plat/inlines.h.in copy platform/linux-generic/include/odp/api/{std_clib.h => plat/std_clib_inlines.h} (52%) copy platform/linux-generic/include/odp/api/{sync.h => plat/sync_inlines.h} (58%) copy test/linux-generic/ring/ring_main.c => platform/linux-generic/odp_byteorder.c (52%) copy test/linux-generic/ring/ring_main.c => platform/linux-generic/odp_std_clib.c (53%) copy test/linux-generic/validation/api/shmem/shmem_odp.h => platform/linux-generic/odp_sync.c (55%) hooks/post-receive --

10 years

[COMMIT] branch master updated. v1.10.1.0-111-ga7f5589

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via a7f5589f81864c71faf18c3d2c3636320cecb1c6 (commit) via aab7083f4ab4fa2ddd8d1a41a554cead3a0055bb (commit) via 66d26268065eaae370b402dc5b006121c6931c5c (commit) via d6260eeec46915b9d90453cc5add6ac0cac74e52 (commit) via ef8b7b691d4902153a71087ed0f4297cdd8809ff (commit) via de19d9847b76c90473c474a831800d383865a4ba (commit) from 2d63311886330e6acddb55674ff7a7b62d711ef5 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit a7f5589f81864c71faf18c3d2c3636320cecb1c6 Author: Petri Savolainen <petri.savolainen(a)nokia.com> Date: Mon Sep 5 13:09:30 2016 +0300 linux-gen: sched: call schedule_release_atomic directly Use local schedule_release_atomic call instead of the API call. Signed-off-by: Petri Savolainen <petri.savolainen(a)nokia.com> Reviewed-and-tested-by: Yi He <yi.he(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index d4eb13f..78982d9 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -201,6 +201,9 @@ static sched_global_t *sched; /* Thread local scheduler context */ __thread sched_local_t sched_local; +/* Function prototypes */ +static inline void schedule_release_context(void); + static void ring_init(sched_ring_t *ring) { odp_atomic_init_u32(&ring->w_head, 0); @@ -377,16 +380,6 @@ static int schedule_init_local(void) return 0; } -static inline void schedule_release_context(void) -{ - if (sched_local.origin_qe != NULL) { - release_order(sched_local.origin_qe, sched_local.order, - sched_local.pool, sched_local.enq_called); - sched_local.origin_qe = NULL; - } else - odp_schedule_release_atomic(); -} - static int schedule_term_local(void) { if (sched_local.num) { @@ -567,6 +560,16 @@ static void schedule_release_ordered(void) } } +static inline void schedule_release_context(void) +{ + if (sched_local.origin_qe != NULL) { + release_order(sched_local.origin_qe, sched_local.order, + sched_local.pool, sched_local.enq_called); + sched_local.origin_qe = NULL; + } else + schedule_release_atomic(); +} + static inline int copy_events(odp_event_t out_ev[], unsigned int max) { int i = 0; commit aab7083f4ab4fa2ddd8d1a41a554cead3a0055bb Author: Petri Savolainen <petri.savolainen(a)nokia.com> Date: Mon Sep 5 13:09:29 2016 +0300 linux-gen: sched: increase max dequeue batch size Increased queue dequeue batch size to improve throughput. Low priority queues have smaller batch size than high/default, so that head of line blocking latency is limited. Signed-off-by: Petri Savolainen <petri.savolainen(a)nokia.com> Reviewed-and-tested-by: Yi He <yi.he(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_schedule_internal.h b/platform/linux-generic/include/odp_schedule_internal.h index 1b6ae93..d6cfdb4 100644 --- a/platform/linux-generic/include/odp_schedule_internal.h +++ b/platform/linux-generic/include/odp_schedule_internal.h @@ -12,7 +12,7 @@ extern "C" { #endif /* Maximum number of dequeues */ -#define MAX_DEQ 4 +#define MAX_DEQ 8 typedef struct { int thr; diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 95ba13f..d4eb13f 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -586,12 +586,13 @@ static inline int copy_events(odp_event_t out_ev[], unsigned int max) * Schedule queues */ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], - unsigned int max_num, unsigned int max_deq) + unsigned int max_num) { - int i, j; + int prio, i; int ret; int id; int offset = 0; + unsigned int max_deq = MAX_DEQ; uint32_t qi; if (sched_local.num) { @@ -619,14 +620,14 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], sched_local.round++; /* Schedule events */ - for (i = 0; i < NUM_PRIO; i++) { + for (prio = 0; prio < NUM_PRIO; prio++) { - if (sched->pri_mask[i] == 0) + if (sched->pri_mask[prio] == 0) continue; id = (sched_local.thr + offset) & (QUEUES_PER_PRIO - 1); - for (j = 0; j < QUEUES_PER_PRIO;) { + for (i = 0; i < QUEUES_PER_PRIO;) { int num; int grp; int ordered; @@ -637,20 +638,20 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], id = 0; /* No queues created for this priority queue */ - if (odp_unlikely((sched->pri_mask[i] & (1 << id)) + if (odp_unlikely((sched->pri_mask[prio] & (1 << id)) == 0)) { - j++; + i++; id++; continue; } /* Get queue index from the priority queue */ - ring = &sched->prio_q[i][id].ring; + ring = &sched->prio_q[prio][id].ring; qi = ring_deq(ring, PRIO_QUEUE_MASK); /* Priority queue empty */ if (qi == RING_EMPTY) { - j++; + i++; id++; continue; } @@ -665,11 +666,16 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], */ ring_enq(ring, PRIO_QUEUE_MASK, qi); - j++; + i++; id++; continue; } + /* Low priorities have smaller batch size to limit + * head of line blocking latency. */ + if (odp_unlikely(prio > ODP_SCHED_PRIO_DEFAULT)) + max_deq = MAX_DEQ / 2; + ordered = sched_cb_queue_is_ordered(qi); /* For ordered queues we want consecutive events to @@ -782,14 +788,14 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], static int schedule_loop(odp_queue_t *out_queue, uint64_t wait, odp_event_t out_ev[], - unsigned int max_num, unsigned int max_deq) + unsigned int max_num) { odp_time_t next, wtime; int first = 1; int ret; while (1) { - ret = do_schedule(out_queue, out_ev, max_num, max_deq); + ret = do_schedule(out_queue, out_ev, max_num); if (ret) break; @@ -820,7 +826,7 @@ static odp_event_t schedule(odp_queue_t *out_queue, uint64_t wait) ev = ODP_EVENT_INVALID; - schedule_loop(out_queue, wait, &ev, 1, MAX_DEQ); + schedule_loop(out_queue, wait, &ev, 1); return ev; } @@ -828,7 +834,7 @@ static odp_event_t schedule(odp_queue_t *out_queue, uint64_t wait) static int schedule_multi(odp_queue_t *out_queue, uint64_t wait, odp_event_t events[], int num) { - return schedule_loop(out_queue, wait, events, num, MAX_DEQ); + return schedule_loop(out_queue, wait, events, num); } static void schedule_pause(void) commit 66d26268065eaae370b402dc5b006121c6931c5c Author: Petri Savolainen <petri.savolainen(a)nokia.com> Date: Mon Sep 5 13:09:28 2016 +0300 linux-gen: sched: pktio poll loop optimization Optimized pktio poll loop if-branches with likely/unlikely hints. Signed-off-by: Petri Savolainen <petri.savolainen(a)nokia.com> Reviewed-and-tested-by: Yi He <yi.he(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 3763a41..95ba13f 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -43,6 +43,9 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && /* Packet input poll cmd queues */ #define PKTIO_CMD_QUEUES 4 +/* Mask for wrapping command queues */ +#define PKTIO_CMD_QUEUE_MASK (PKTIO_CMD_QUEUES - 1) + /* Maximum number of packet input queues per command */ #define MAX_PKTIN 16 @@ -94,6 +97,11 @@ ODP_STATIC_ASSERT(ODP_VAL_IS_POWER_2(PRIO_QUEUE_RING_SIZE), ODP_STATIC_ASSERT(ODP_VAL_IS_POWER_2(PKTIO_RING_SIZE), "pktio_ring_size_is_not_power_of_two"); +/* Number of commands queues must be power of two, so that PKTIO_CMD_QUEUE_MASK + * can be used. */ +ODP_STATIC_ASSERT(ODP_VAL_IS_POWER_2(PKTIO_CMD_QUEUES), + "pktio_cmd_queues_is_not_power_of_two"); + /* Mask of queues per priority */ typedef uint8_t pri_mask_t; @@ -452,7 +460,7 @@ static void schedule_destroy_queue(uint32_t queue_index) static int poll_cmd_queue_idx(int pktio_index, int pktin_idx) { - return (PKTIO_CMD_QUEUES - 1) & (pktio_index ^ pktin_idx); + return PKTIO_CMD_QUEUE_MASK & (pktio_index ^ pktin_idx); } static inline pktio_cmd_t *alloc_pktio_cmd(void) @@ -725,28 +733,29 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], * have to do full iteration to avoid packet input starvation when * there are less threads than command queues. */ - id = sched_local.thr & (PKTIO_CMD_QUEUES - 1); + id = sched_local.thr & PKTIO_CMD_QUEUE_MASK; - for (i = 0; i < PKTIO_CMD_QUEUES; i++, id++) { + for (i = 0; i < PKTIO_CMD_QUEUES; i++, id = ((id + 1) & + PKTIO_CMD_QUEUE_MASK)) { + sched_ring_t *ring; uint32_t cmd_index; pktio_cmd_t *cmd; - if (id == PKTIO_CMD_QUEUES) - id = 0; - - if (sched->num_pktio_cmd[id] == 0) + if (odp_unlikely(sched->num_pktio_cmd[id] == 0)) continue; - cmd_index = ring_deq(&sched->pktio_q[id].ring, PKTIO_RING_MASK); + ring = &sched->pktio_q[id].ring; + cmd_index = ring_deq(ring, PKTIO_RING_MASK); - if (cmd_index == RING_EMPTY) + if (odp_unlikely(cmd_index == RING_EMPTY)) continue; cmd = &sched->pktio_cmd[cmd_index]; /* Poll packet input */ - if (sched_cb_pktin_poll(cmd->pktio_index, - cmd->num_pktin, cmd->pktin)){ + if (odp_unlikely(sched_cb_pktin_poll(cmd->pktio_index, + cmd->num_pktin, + cmd->pktin))){ /* Pktio stopped or closed. Remove poll command and call * stop_finalize when all commands of the pktio has * been removed. */ @@ -757,8 +766,7 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], free_pktio_cmd(cmd); } else { /* Continue scheduling the pktio */ - ring_enq(&sched->pktio_q[id].ring, PKTIO_RING_MASK, - cmd_index); + ring_enq(ring, PKTIO_RING_MASK, cmd_index); /* Do not iterate through all pktin poll command queues * every time. */ commit d6260eeec46915b9d90453cc5add6ac0cac74e52 Author: Petri Savolainen <petri.savolainen(a)nokia.com> Date: Mon Sep 5 13:09:27 2016 +0300 linux-gen: sched: increase number of pktin Increased maximum number of pktin queues supported by the scheduler. Signed-off-by: Petri Savolainen <petri.savolainen(a)nokia.com> Reviewed-and-tested-by: Yi He <yi.he(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 07c21d5..3763a41 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -44,7 +44,7 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && #define PKTIO_CMD_QUEUES 4 /* Maximum number of packet input queues per command */ -#define MAX_PKTIN 8 +#define MAX_PKTIN 16 /* Maximum number of packet IO interfaces */ #define NUM_PKTIO ODP_CONFIG_PKTIO_ENTRIES commit ef8b7b691d4902153a71087ed0f4297cdd8809ff Author: Petri Savolainen <petri.savolainen(a)nokia.com> Date: Mon Sep 5 13:09:26 2016 +0300 linux-gen: sched: ring based pktio command queues Implement scheduler pktio poll command queues with a ring based data structure instead of odp_queue_t queues. Also scheduler initialization and termination is simplified as there is no more dependency to pool, buffer and queue APIs. Signed-off-by: Petri Savolainen <petri.savolainen(a)nokia.com> Reviewed-and-tested-by: Yi He <yi.he(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 83c81e4..07c21d5 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -8,10 +8,7 @@ #include <odp/api/schedule.h> #include <odp_schedule_if.h> #include <odp/api/align.h> -#include <odp/api/queue.h> #include <odp/api/shared_memory.h> -#include <odp/api/buffer.h> -#include <odp/api/pool.h> #include <odp_internal.h> #include <odp_debug_internal.h> #include <odp/api/thread.h> @@ -27,10 +24,6 @@ #include <odp_schedule_ordered_internal.h> #include <odp/api/sync.h> -#ifdef _ODP_PKTIO_IPC -#include <odp_pool_internal.h> -#endif - /* Number of priority levels */ #define NUM_PRIO 8 @@ -48,7 +41,7 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && #define QUEUES_PER_PRIO 4 /* Packet input poll cmd queues */ -#define POLL_CMD_QUEUES 4 +#define PKTIO_CMD_QUEUES 4 /* Maximum number of packet input queues per command */ #define MAX_PKTIN 8 @@ -56,6 +49,24 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && /* Maximum number of packet IO interfaces */ #define NUM_PKTIO ODP_CONFIG_PKTIO_ENTRIES +/* Maximum number of pktio poll commands */ +#define NUM_PKTIO_CMD (MAX_PKTIN * NUM_PKTIO) + +/* Not a valid poll command */ +#define PKTIO_CMD_INVALID ((uint32_t)-1) + +/* Pktio command is free */ +#define PKTIO_CMD_FREE PKTIO_CMD_INVALID + +/* Packet IO poll queue ring size. In worst case, all pktios have all pktins + * enabled and one poll command is created per pktin queue. The ring size must + * be larger than or equal to NUM_PKTIO_CMD / PKTIO_CMD_QUEUES, so that it can + * hold all poll commands in the worst case. */ +#define PKTIO_RING_SIZE (NUM_PKTIO_CMD / PKTIO_CMD_QUEUES) + +/* Mask for wrapping around pktio poll command index */ +#define PKTIO_RING_MASK (PKTIO_RING_SIZE - 1) + /* Priority queue ring size. In worst case, all event queues are scheduled * queues and have the same priority. The ring size must be larger than or * equal to ODP_CONFIG_QUEUES / QUEUES_PER_PRIO, so that it can hold all @@ -79,6 +90,10 @@ ODP_STATIC_ASSERT(ODP_VAL_IS_POWER_2(ODP_CONFIG_QUEUES), ODP_STATIC_ASSERT(ODP_VAL_IS_POWER_2(PRIO_QUEUE_RING_SIZE), "Ring_size_is_not_power_of_two"); +/* Ring size must be power of two, so that PKTIO_RING_MASK can be used. */ +ODP_STATIC_ASSERT(ODP_VAL_IS_POWER_2(PKTIO_RING_SIZE), + "pktio_ring_size_is_not_power_of_two"); + /* Mask of queues per priority */ typedef uint8_t pri_mask_t; @@ -116,6 +131,24 @@ typedef struct { } prio_queue_t ODP_ALIGNED_CACHE; +/* Packet IO queue */ +typedef struct { + /* Ring header */ + sched_ring_t ring; + + /* Ring data: pktio poll command indexes */ + uint32_t cmd_index[PKTIO_RING_SIZE]; + +} pktio_queue_t ODP_ALIGNED_CACHE; + +/* Packet IO poll command */ +typedef struct { + int pktio_index; + int num_pktin; + int pktin[MAX_PKTIN]; + uint32_t cmd_index; +} pktio_cmd_t; + typedef struct { pri_mask_t pri_mask[NUM_PRIO]; odp_spinlock_t mask_lock; @@ -123,12 +156,15 @@ typedef struct { prio_queue_t prio_q[NUM_PRIO][QUEUES_PER_PRIO]; odp_spinlock_t poll_cmd_lock; - struct { - odp_queue_t queue; - uint16_t num; - } poll_cmd[POLL_CMD_QUEUES]; + /* Number of commands in a command queue */ + uint16_t num_pktio_cmd[PKTIO_CMD_QUEUES]; + + /* Packet IO command queues */ + pktio_queue_t pktio_q[PKTIO_CMD_QUEUES]; + + /* Packet IO poll commands */ + pktio_cmd_t pktio_cmd[NUM_PKTIO_CMD]; - odp_pool_t pool; odp_shm_t shm; uint32_t pri_count[NUM_PRIO][QUEUES_PER_PRIO]; @@ -145,31 +181,12 @@ typedef struct { } queue[ODP_CONFIG_QUEUES]; struct { - int num; + /* Number of active commands for a pktio interface */ + int num_cmd; } pktio[NUM_PKTIO]; } sched_global_t; -/* Schedule command */ -typedef struct { - int cmd; - - union { - struct { - uint32_t queue_index; - }; - - struct { - int pktio_index; - int num; - int index[MAX_PKTIN]; - }; - }; -} sched_cmd_t; - -#define SCHED_CMD_DEQUEUE 0 -#define SCHED_CMD_POLL_PKTIN 1 - /* Global scheduler context */ static sched_global_t *sched; @@ -253,10 +270,7 @@ static void sched_local_init(void) static int schedule_init_global(void) { odp_shm_t shm; - odp_pool_t pool; int i, j; - odp_pool_param_t params; - int num_cmd; ODP_DBG("Schedule init ... "); @@ -273,27 +287,6 @@ static int schedule_init_global(void) memset(sched, 0, sizeof(sched_global_t)); - /* Number of schedule commands. - * One per scheduled queue and packet interface */ - num_cmd = sched_cb_num_queues() + sched_cb_num_pktio(); - - odp_pool_param_init(&params); - params.buf.size = sizeof(sched_cmd_t); - params.buf.align = 0; - params.buf.num = num_cmd; - params.type = ODP_POOL_BUFFER; - -#ifdef _ODP_PKTIO_IPC - pool = _pool_create("odp_sched_pool", &params, 0); -#else - pool = odp_pool_create("odp_sched_pool", &params); -#endif - if (pool == ODP_POOL_INVALID) { - ODP_ERR("Schedule init: Pool create failed.\n"); - return -1; - } - - sched->pool = pool; sched->shm = shm; odp_spinlock_init(&sched->mask_lock); @@ -310,23 +303,16 @@ static int schedule_init_global(void) } odp_spinlock_init(&sched->poll_cmd_lock); - for (i = 0; i < POLL_CMD_QUEUES; i++) { - odp_queue_t queue; - char name[] = "odp_poll_cmd_YY"; - - name[13] = '0' + i / 10; - name[14] = '0' + i - 10 * (i / 10); - - queue = odp_queue_create(name, NULL); + for (i = 0; i < PKTIO_CMD_QUEUES; i++) { + ring_init(&sched->pktio_q[i].ring); - if (queue == ODP_QUEUE_INVALID) { - ODP_ERR("Sched init: Queue create failed.\n"); - return -1; - } - - sched->poll_cmd[i].queue = queue; + for (j = 0; j < PKTIO_RING_SIZE; j++) + sched->pktio_q[i].cmd_index[j] = PKTIO_CMD_INVALID; } + for (i = 0; i < NUM_PKTIO_CMD; i++) + sched->pktio_cmd[i].cmd_index = PKTIO_CMD_FREE; + odp_spinlock_init(&sched->grp_lock); for (i = 0; i < NUM_SCHED_GRPS; i++) { @@ -346,7 +332,6 @@ static int schedule_term_global(void) int ret = 0; int rc = 0; int i, j; - odp_event_t ev; for (i = 0; i < NUM_PRIO; i++) { for (j = 0; j < QUEUES_PER_PRIO; j++) { @@ -369,23 +354,6 @@ static int schedule_term_global(void) } } - for (i = 0; i < POLL_CMD_QUEUES; i++) { - odp_queue_t queue = sched->poll_cmd[i].queue; - - while ((ev = odp_queue_deq(queue)) != ODP_EVENT_INVALID) - odp_event_free(ev); - - if (odp_queue_destroy(queue)) { - ODP_ERR("Poll cmd queue destroy failed\n"); - rc = -1; - } - } - - if (odp_pool_destroy(sched->pool) != 0) { - ODP_ERR("Pool destroy fail.\n"); - rc = -1; - } - ret = odp_shm_free(sched->shm); if (ret < 0) { ODP_ERR("Shm free failed for odp_scheduler"); @@ -482,60 +450,83 @@ static void schedule_destroy_queue(uint32_t queue_index) sched->queue[queue_index].queue_per_prio = 0; } -static int poll_cmd_queue_idx(int pktio_index, int in_queue_idx) +static int poll_cmd_queue_idx(int pktio_index, int pktin_idx) { - return (POLL_CMD_QUEUES - 1) & (pktio_index ^ in_queue_idx); + return (PKTIO_CMD_QUEUES - 1) & (pktio_index ^ pktin_idx); } -static void schedule_pktio_start(int pktio_index, int num_in_queue, - int in_queue_idx[]) +static inline pktio_cmd_t *alloc_pktio_cmd(void) +{ + int i; + pktio_cmd_t *cmd = NULL; + + odp_spinlock_lock(&sched->poll_cmd_lock); + + /* Find next free command */ + for (i = 0; i < NUM_PKTIO_CMD; i++) { + if (sched->pktio_cmd[i].cmd_index == PKTIO_CMD_FREE) { + cmd = &sched->pktio_cmd[i]; + cmd->cmd_index = i; + break; + } + } + + odp_spinlock_unlock(&sched->poll_cmd_lock); + + return cmd; +} + +static inline void free_pktio_cmd(pktio_cmd_t *cmd) +{ + odp_spinlock_lock(&sched->poll_cmd_lock); + + cmd->cmd_index = PKTIO_CMD_FREE; + + odp_spinlock_unlock(&sched->poll_cmd_lock); +} + +static void schedule_pktio_start(int pktio_index, int num_pktin, + int pktin_idx[]) { - odp_buffer_t buf; - sched_cmd_t *sched_cmd; - odp_queue_t queue; int i, idx; + pktio_cmd_t *cmd; - if (num_in_queue > MAX_PKTIN) + if (num_pktin > MAX_PKTIN) ODP_ABORT("Too many input queues for scheduler\n"); - sched->pktio[pktio_index].num = num_in_queue; + sched->pktio[pktio_index].num_cmd = num_pktin; /* Create a pktio poll command per queue */ - for (i = 0; i < num_in_queue; i++) { - buf = odp_buffer_alloc(sched->pool); + for (i = 0; i < num_pktin; i++) { - if (buf == ODP_BUFFER_INVALID) - ODP_ABORT("Sched pool empty\n"); + cmd = alloc_pktio_cmd(); - sched_cmd = odp_buffer_addr(buf); - sched_cmd->cmd = SCHED_CMD_POLL_PKTIN; - sched_cmd->pktio_index = pktio_index; - sched_cmd->num = 1; - sched_cmd->index[0] = in_queue_idx[i]; + if (cmd == NULL) + ODP_ABORT("Scheduler out of pktio commands\n"); - idx = poll_cmd_queue_idx(pktio_index, in_queue_idx[i]); + idx = poll_cmd_queue_idx(pktio_index, pktin_idx[i]); odp_spinlock_lock(&sched->poll_cmd_lock); - sched->poll_cmd[idx].num++; + sched->num_pktio_cmd[idx]++; odp_spinlock_unlock(&sched->poll_cmd_lock); - queue = sched->poll_cmd[idx].queue; - - if (odp_queue_enq(queue, odp_buffer_to_event(buf))) - ODP_ABORT("schedule_pktio_start failed\n"); + cmd->pktio_index = pktio_index; + cmd->num_pktin = 1; + cmd->pktin[0] = pktin_idx[i]; + ring_enq(&sched->pktio_q[idx].ring, PKTIO_RING_MASK, + cmd->cmd_index); } } -static int schedule_pktio_stop(sched_cmd_t *sched_cmd) +static int schedule_pktio_stop(int pktio_index, int first_pktin) { int num; - int pktio_index = sched_cmd->pktio_index; - int idx = poll_cmd_queue_idx(pktio_index, sched_cmd->index[0]); + int idx = poll_cmd_queue_idx(pktio_index, first_pktin); odp_spinlock_lock(&sched->poll_cmd_lock); - sched->poll_cmd[idx].num--; - sched->pktio[pktio_index].num--; - num = sched->pktio[pktio_index].num; + sched->num_pktio_cmd[idx]--; + sched->pktio[pktio_index].num_cmd--; + num = sched->pktio[pktio_index].num_cmd; odp_spinlock_unlock(&sched->poll_cmd_lock); return num; @@ -592,9 +583,6 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], int i, j; int ret; int id; - odp_event_t ev; - odp_buffer_t buf; - sched_cmd_t *sched_cmd; int offset = 0; uint32_t qi; @@ -737,47 +725,40 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], * have to do full iteration to avoid packet input starvation when * there are less threads than command queues. */ - id = sched_local.thr & (POLL_CMD_QUEUES - 1); + id = sched_local.thr & (PKTIO_CMD_QUEUES - 1); - for (i = 0; i < POLL_CMD_QUEUES; i++, id++) { - odp_queue_t cmd_queue; - int pktio_index; + for (i = 0; i < PKTIO_CMD_QUEUES; i++, id++) { + uint32_t cmd_index; + pktio_cmd_t *cmd; - if (id == POLL_CMD_QUEUES) + if (id == PKTIO_CMD_QUEUES) id = 0; - if (sched->poll_cmd[id].num == 0) + if (sched->num_pktio_cmd[id] == 0) continue; - cmd_queue = sched->poll_cmd[id].queue; - ev = odp_queue_deq(cmd_queue); + cmd_index = ring_deq(&sched->pktio_q[id].ring, PKTIO_RING_MASK); - if (ev == ODP_EVENT_INVALID) + if (cmd_index == RING_EMPTY) continue; - buf = odp_buffer_from_event(ev); - sched_cmd = odp_buffer_addr(buf); - - if (sched_cmd->cmd != SCHED_CMD_POLL_PKTIN) - ODP_ABORT("Bad poll command\n"); - - pktio_index = sched_cmd->pktio_index; + cmd = &sched->pktio_cmd[cmd_index]; /* Poll packet input */ - if (sched_cb_pktin_poll(pktio_index, - sched_cmd->num, - sched_cmd->index)) { + if (sched_cb_pktin_poll(cmd->pktio_index, + cmd->num_pktin, cmd->pktin)){ /* Pktio stopped or closed. Remove poll command and call * stop_finalize when all commands of the pktio has * been removed. */ - if (schedule_pktio_stop(sched_cmd) == 0) - sched_cb_pktio_stop_finalize(pktio_index); + if (schedule_pktio_stop(cmd->pktio_index, + cmd->pktin[0]) == 0) + sched_cb_pktio_stop_finalize(cmd->pktio_index); - odp_buffer_free(buf); + free_pktio_cmd(cmd); } else { /* Continue scheduling the pktio */ - if (odp_queue_enq(cmd_queue, ev)) - ODP_ABORT("Poll command enqueue failed\n"); + ring_enq(&sched->pktio_q[id].ring, PKTIO_RING_MASK, + cmd_index); /* Do not iterate through all pktin poll command queues * every time. */ commit de19d9847b76c90473c474a831800d383865a4ba Author: Petri Savolainen <petri.savolainen(a)nokia.com> Date: Mon Sep 5 13:09:25 2016 +0300 linux-gen: sched: ring based priority queues Implemented scheduler priority queues with a ring based data structure instead of odp_queue_t queues. This enables performance optimization since a ring of indexes is more efficient to access than pointers in a linked list. Ring operations maybe further optimized with another lockfree algorithm. Signed-off-by: Petri Savolainen <petri.savolainen(a)nokia.com> Reviewed-and-tested-by: Yi He <yi.he(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_schedule_internal.h b/platform/linux-generic/include/odp_schedule_internal.h index 4a04b15..1b6ae93 100644 --- a/platform/linux-generic/include/odp_schedule_internal.h +++ b/platform/linux-generic/include/odp_schedule_internal.h @@ -22,8 +22,7 @@ typedef struct { uint16_t round; uint16_t prefer_offset; uint16_t pktin_polls; - odp_queue_t pri_queue; - odp_event_t cmd_ev; + uint32_t queue_index; odp_queue_t queue; odp_event_t ev_stash[MAX_DEQ]; void *origin_qe; diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index e08de54..83c81e4 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -20,9 +20,13 @@ #include <odp/api/hints.h> #include <odp/api/cpu.h> #include <odp/api/thrmask.h> +#include <odp/api/atomic.h> #include <odp_config_internal.h> +#include <odp_align_internal.h> #include <odp_schedule_internal.h> #include <odp_schedule_ordered_internal.h> +#include <odp/api/sync.h> + #ifdef _ODP_PKTIO_IPC #include <odp_pool_internal.h> #endif @@ -52,6 +56,29 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && /* Maximum number of packet IO interfaces */ #define NUM_PKTIO ODP_CONFIG_PKTIO_ENTRIES +/* Priority queue ring size. In worst case, all event queues are scheduled + * queues and have the same priority. The ring size must be larger than or + * equal to ODP_CONFIG_QUEUES / QUEUES_PER_PRIO, so that it can hold all + * queues in the worst case. */ +#define PRIO_QUEUE_RING_SIZE (ODP_CONFIG_QUEUES / QUEUES_PER_PRIO) + +/* Mask for wrapping around priority queue index */ +#define PRIO_QUEUE_MASK (PRIO_QUEUE_RING_SIZE - 1) + +/* Priority queue empty, not a valid queue index. */ +#define PRIO_QUEUE_EMPTY ((uint32_t)-1) + +/* Ring empty, not a valid index. */ +#define RING_EMPTY ((uint32_t)-1) + +/* For best performance, the number of queues should be a power of two. */ +ODP_STATIC_ASSERT(ODP_VAL_IS_POWER_2(ODP_CONFIG_QUEUES), + "Number_of_queues_is_not_power_of_two"); + +/* Ring size must be power of two, so that MAX_QUEUE_IDX_MASK can be used. */ +ODP_STATIC_ASSERT(ODP_VAL_IS_POWER_2(PRIO_QUEUE_RING_SIZE), + "Ring_size_is_not_power_of_two"); + /* Mask of queues per priority */ typedef uint8_t pri_mask_t; @@ -61,11 +88,40 @@ ODP_STATIC_ASSERT((8 * sizeof(pri_mask_t)) >= QUEUES_PER_PRIO, /* Start of named groups in group mask arrays */ #define SCHED_GROUP_NAMED (ODP_SCHED_GROUP_CONTROL + 1) +/* Scheduler ring + * + * Ring stores head and tail counters. Ring indexes are formed from these + * counters with a mask (mask = ring_size - 1), which requires that ring size + * must be a power of two. */ +typedef struct { + /* Writer head and tail */ + odp_atomic_u32_t w_head; + odp_atomic_u32_t w_tail; + uint8_t pad[ODP_CACHE_LINE_SIZE - (2 * sizeof(odp_atomic_u32_t))]; + + /* Reader head and tail */ + odp_atomic_u32_t r_head; + odp_atomic_u32_t r_tail; + + uint32_t data[0]; +} sched_ring_t ODP_ALIGNED_CACHE; + +/* Priority queue */ +typedef struct { + /* Ring header */ + sched_ring_t ring; + + /* Ring data: queue indexes */ + uint32_t queue_index[PRIO_QUEUE_RING_SIZE]; + +} prio_queue_t ODP_ALIGNED_CACHE; + typedef struct { - odp_queue_t pri_queue[NUM_PRIO][QUEUES_PER_PRIO]; pri_mask_t pri_mask[NUM_PRIO]; odp_spinlock_t mask_lock; + prio_queue_t prio_q[NUM_PRIO][QUEUES_PER_PRIO]; + odp_spinlock_t poll_cmd_lock; struct { odp_queue_t queue; @@ -84,9 +140,8 @@ typedef struct { } sched_grp[NUM_SCHED_GRPS]; struct { - odp_event_t cmd_ev; - odp_queue_t pri_queue; int prio; + int queue_per_prio; } queue[ODP_CONFIG_QUEUES]; struct { @@ -121,14 +176,78 @@ static sched_global_t *sched; /* Thread local scheduler context */ __thread sched_local_t sched_local; +static void ring_init(sched_ring_t *ring) +{ + odp_atomic_init_u32(&ring->w_head, 0); + odp_atomic_init_u32(&ring->w_tail, 0); + odp_atomic_init_u32(&ring->r_head, 0); + odp_atomic_init_u32(&ring->r_tail, 0); +} + +/* Dequeue data from the ring head */ +static inline uint32_t ring_deq(sched_ring_t *ring, uint32_t mask) +{ + uint32_t head, tail, new_head; + uint32_t data; + + head = odp_atomic_load_u32(&ring->r_head); + + /* Move reader head. This thread owns data at the new head. */ + do { + tail = odp_atomic_load_u32(&ring->w_tail); + + if (head == tail) + return RING_EMPTY; + + new_head = head + 1; + + } while (odp_unlikely(odp_atomic_cas_acq_u32(&ring->r_head, &head, + new_head) == 0)); + + /* Read queue index */ + data = ring->data[new_head & mask]; + + /* Wait until other readers have updated the tail */ + while (odp_unlikely(odp_atomic_load_acq_u32(&ring->r_tail) != head)) + odp_cpu_pause(); + + /* Now update the reader tail */ + odp_atomic_store_rel_u32(&ring->r_tail, new_head); + + return data; +} + +/* Enqueue data into the ring tail */ +static inline void ring_enq(sched_ring_t *ring, uint32_t mask, uint32_t data) +{ + uint32_t old_head, new_head; + + /* Reserve a slot in the ring for writing */ + old_head = odp_atomic_fetch_inc_u32(&ring->w_head); + new_head = old_head + 1; + + /* Ring is full. Wait for the last reader to finish. */ + while (odp_unlikely(odp_atomic_load_acq_u32(&ring->r_tail) == new_head)) + odp_cpu_pause(); + + /* Write data */ + ring->data[new_head & mask] = data; + + /* Wait until other writers have updated the tail */ + while (odp_unlikely(odp_atomic_load_acq_u32(&ring->w_tail) != old_head)) + odp_cpu_pause(); + + /* Now update the writer tail */ + odp_atomic_store_rel_u32(&ring->w_tail, new_head); +} + static void sched_local_init(void) { memset(&sched_local, 0, sizeof(sched_local_t)); sched_local.thr = odp_thread_id(); - sched_local.pri_queue = ODP_QUEUE_INVALID; sched_local.queue = ODP_QUEUE_INVALID; - sched_local.cmd_ev = ODP_EVENT_INVALID; + sched_local.queue_index = PRIO_QUEUE_EMPTY; } static int schedule_init_global(void) @@ -179,25 +298,14 @@ static int schedule_init_global(void) odp_spinlock_init(&sched->mask_lock); for (i = 0; i < NUM_PRIO; i++) { - odp_queue_t queue; - char name[] = "odp_priXX_YY"; - - name[7] = '0' + i / 10; - name[8] = '0' + i - 10*(i / 10); - for (j = 0; j < QUEUES_PER_PRIO; j++) { - name[10] = '0' + j / 10; - name[11] = '0' + j - 10*(j / 10); - - queue = odp_queue_create(name, NULL); + int k; - if (queue == ODP_QUEUE_INVALID) { - ODP_ERR("Sched init: Queue create failed.\n"); - return -1; - } + ring_init(&sched->prio_q[i][j].ring); - sched->pri_queue[i][j] = queue; - sched->pri_mask[i] = 0; + for (k = 0; k < PRIO_QUEUE_RING_SIZE; k++) + sched->prio_q[i][j].queue_index[k] = + PRIO_QUEUE_EMPTY; } } @@ -228,11 +336,6 @@ static int schedule_init_global(void) odp_thrmask_setall(&sched->mask_all); - for (i = 0; i < ODP_CONFIG_QUEUES; i++) { - sched->queue[i].cmd_ev = ODP_EVENT_INVALID; - sched->queue[i].pri_queue = ODP_QUEUE_INVALID; - } - ODP_DBG("done\n"); return 0; @@ -247,21 +350,14 @@ static int schedule_term_global(void) for (i = 0; i < NUM_PRIO; i++) { for (j = 0; j < QUEUES_PER_PRIO; j++) { - odp_queue_t pri_q; - - pri_q = sched->pri_queue[i][j]; + sched_ring_t *ring = &sched->prio_q[i][j].ring; + uint32_t qi; - while ((ev = odp_queue_deq(pri_q)) != - ODP_EVENT_INVALID) { - odp_buffer_t buf; - sched_cmd_t *sched_cmd; - uint32_t qi; + while ((qi = ring_deq(ring, PRIO_QUEUE_MASK)) != + RING_EMPTY) { odp_event_t events[1]; int num; - buf = odp_buffer_from_event(ev); - sched_cmd = odp_buffer_addr(buf); - qi = sched_cmd->queue_index; num = sched_cb_queue_deq_multi(qi, events, 1); if (num < 0) @@ -270,11 +366,6 @@ static int schedule_term_global(void) if (num > 0) ODP_ERR("Queue not empty\n"); } - - if (odp_queue_destroy(pri_q)) { - ODP_ERR("Pri queue destroy fail.\n"); - rc = -1; - } } } @@ -331,19 +422,17 @@ static int schedule_term_local(void) return 0; } -static inline int pri_id_queue(uint32_t queue_index) +static inline int queue_per_prio(uint32_t queue_index) { return ((QUEUES_PER_PRIO - 1) & queue_index); } -static odp_queue_t pri_set(int id, int prio) +static void pri_set(int id, int prio) { odp_spinlock_lock(&sched->mask_lock); sched->pri_mask[prio] |= 1 << id; sched->pri_count[prio][id]++; odp_spinlock_unlock(&sched->mask_lock); - - return sched->pri_queue[prio][id]; } static void pri_clr(int id, int prio) @@ -359,38 +448,27 @@ static void pri_clr(int id, int prio) odp_spinlock_unlock(&sched->mask_lock); } -static odp_queue_t pri_set_queue(uint32_t queue_index, int prio) +static void pri_set_queue(uint32_t queue_index, int prio) { - int id = pri_id_queue(queue_index); + int id = queue_per_prio(queue_index); return pri_set(id, prio); } static void pri_clr_queue(uint32_t queue_index, int prio) { - int id = pri_id_queue(queue_index); + int id = queue_per_prio(queue_index); pri_clr(id, prio); } static int schedule_init_queue(uint32_t queue_index, const odp_schedule_param_t *sched_param) { - odp_buffer_t buf; - sched_cmd_t *sched_cmd; int prio = sched_param->prio; - buf = odp_buffer_alloc(sched->pool); - - if (buf == ODP_BUFFER_INVALID) - return -1; - - sched_cmd = odp_buffer_addr(buf); - sched_cmd->cmd = SCHED_CMD_DEQUEUE; - sched_cmd->queue_index = queue_index; - - sched->queue[queue_index].cmd_ev = odp_buffer_to_event(buf); - sched->queue[queue_index].pri_queue = pri_set_queue(queue_index, prio); + pri_set_queue(queue_index, prio); sched->queue[queue_index].prio = prio; + sched->queue[queue_index].queue_per_prio = queue_per_prio(queue_index); return 0; } @@ -399,13 +477,9 @@ static void schedule_destroy_queue(uint32_t queue_index) { int prio = sched->queue[queue_index].prio; - odp_event_free(sched->queue[queue_index].cmd_ev); - pri_clr_queue(queue_index, prio); - - sched->queue[queue_index].cmd_ev = ODP_EVENT_INVALID; - sched->queue[queue_index].pri_queue = ODP_QUEUE_INVALID; - sched->queue[queue_index].prio = 0; + sched->queue[queue_index].prio = 0; + sched->queue[queue_index].queue_per_prio = 0; } static int poll_cmd_queue_idx(int pktio_index, int in_queue_idx) @@ -469,12 +543,16 @@ static int schedule_pktio_stop(sched_cmd_t *sched_cmd) static void schedule_release_atomic(void) { - if (sched_local.pri_queue != ODP_QUEUE_INVALID && - sched_local.num == 0) { + uint32_t qi = sched_local.queue_index; + + if (qi != PRIO_QUEUE_EMPTY && sched_local.num == 0) { + int prio = sched->queue[qi].prio; + int queue_per_prio = sched->queue[qi].queue_per_prio; + sched_ring_t *ring = &sched->prio_q[prio][queue_per_prio].ring; + /* Release current atomic queue */ - if (odp_queue_enq(sched_local.pri_queue, sched_local.cmd_ev)) - ODP_ABORT("odp_schedule_release_atomic failed\n"); - sched_local.pri_queue = ODP_QUEUE_INVALID; + ring_enq(ring, PRIO_QUEUE_MASK, qi); + sched_local.queue_index = PRIO_QUEUE_EMPTY; } } @@ -553,11 +631,11 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], id = (sched_local.thr + offset) & (QUEUES_PER_PRIO - 1); for (j = 0; j < QUEUES_PER_PRIO;) { - odp_queue_t pri_q; int num; int grp; int ordered; odp_queue_t handle; + sched_ring_t *ring; if (id >= QUEUES_PER_PRIO) id = 0; @@ -570,20 +648,18 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], continue; } - pri_q = sched->pri_queue[i][id]; - ev = odp_queue_deq(pri_q); + /* Get queue index from the priority queue */ + ring = &sched->prio_q[i][id].ring; + qi = ring_deq(ring, PRIO_QUEUE_MASK); /* Priority queue empty */ - if (ev == ODP_EVENT_INVALID) { + if (qi == RING_EMPTY) { j++; id++; continue; } - buf = odp_buffer_from_event(ev); - sched_cmd = odp_buffer_addr(buf); - qi = sched_cmd->queue_index; - grp = sched_cb_queue_grp(qi); + grp = sched_cb_queue_grp(qi); if (grp > ODP_SCHED_GROUP_ALL && !odp_thrmask_isset(&sched->sched_grp[grp].mask, @@ -591,8 +667,7 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], /* This thread is not eligible for work from * this queue, so continue scheduling it. */ - if (odp_queue_enq(pri_q, ev)) - ODP_ABORT("schedule failed\n"); + ring_enq(ring, PRIO_QUEUE_MASK, qi); j++; id++; @@ -632,19 +707,16 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], if (ordered) { /* Continue scheduling ordered queues */ - if (odp_queue_enq(pri_q, ev)) - ODP_ABORT("schedule failed\n"); + ring_enq(ring, PRIO_QUEUE_MASK, qi); /* Cache order info about this event */ cache_order_info(qi); } else if (sched_cb_queue_is_atomic(qi)) { /* Hold queue during atomic access */ - sched_local.pri_queue = pri_q; - sched_local.cmd_ev = ev; + sched_local.queue_index = qi; } else { /* Continue scheduling the queue */ - if (odp_queue_enq(pri_q, ev)) - ODP_ABORT("schedule failed\n"); + ring_enq(ring, PRIO_QUEUE_MASK, qi); } /* Output the source queue handle */ @@ -969,11 +1041,14 @@ static void schedule_prefetch(int num ODP_UNUSED) static int schedule_sched_queue(uint32_t queue_index) { - odp_queue_t pri_queue = sched->queue[queue_index].pri_queue; - odp_event_t cmd_ev = sched->queue[queue_index].cmd_ev; + int prio = sched->queue[queue_index].prio; + int queue_per_prio = sched->queue[queue_index].queue_per_prio; + sched_ring_t *ring = &sched->prio_q[prio][queue_per_prio].ring; sched_local.ignore_ordered_context = 1; - return odp_queue_enq(pri_queue, cmd_ev); + + ring_enq(ring, PRIO_QUEUE_MASK, queue_index); + return 0; } static int schedule_num_grps(void) ----------------------------------------------------------------------- Summary of changes: .../linux-generic/include/odp_schedule_internal.h | 5 +- platform/linux-generic/odp_schedule.c | 599 ++++++++++++--------- 2 files changed, 338 insertions(+), 266 deletions(-) hooks/post-receive --

10 years

[COMMIT] branch master updated. v1.10.1.0-105-g2d63311

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 2d63311886330e6acddb55674ff7a7b62d711ef5 (commit) from d96981e81ecbfc54ac8832d3ed2c32f50eaaf9a1 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 2d63311886330e6acddb55674ff7a7b62d711ef5 Author: Petri Savolainen <petri.savolainen(a)nokia.com> Date: Mon Sep 5 13:29:50 2016 +0300 test: l2fwd: remove wait time from sched mode Application requested 100ms timeout for no reason in the scheduler mode. For example on odp-linux, performance is improved about 10% when the timeout is removed (as time keeping creates system calls). Signed-off-by: Petri Savolainen <petri.savolainen(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/performance/odp_l2fwd.c b/test/common_plat/performance/odp_l2fwd.c index 418382d..4020dea 100644 --- a/test/common_plat/performance/odp_l2fwd.c +++ b/test/common_plat/performance/odp_l2fwd.c @@ -305,7 +305,6 @@ static int run_worker_sched_mode(void *arg) odp_packet_t pkt_tbl[MAX_PKT_BURST]; int pkts; int thr; - uint64_t wait; int dst_idx; int thr_idx; int i; @@ -344,14 +343,13 @@ static int run_worker_sched_mode(void *arg) odp_barrier_wait(&barrier); - wait = odp_schedule_wait_time(ODP_TIME_MSEC_IN_NS * 100); - /* Loop packets */ while (!exit_threads) { int sent; unsigned tx_drops; - pkts = odp_schedule_multi(NULL, wait, ev_tbl, MAX_PKT_BURST); + pkts = odp_schedule_multi(NULL, ODP_SCHED_NO_WAIT, ev_tbl, + MAX_PKT_BURST); if (pkts <= 0) continue; ----------------------------------------------------------------------- Summary of changes: test/common_plat/performance/odp_l2fwd.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) hooks/post-receive --

10 years

[COMMIT] branch master updated. v1.10.1.0-104-gd96981e

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via d96981e81ecbfc54ac8832d3ed2c32f50eaaf9a1 (commit) from 7cd13f48c935a88d56fd7d393fea6f02e5b0365a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit d96981e81ecbfc54ac8832d3ed2c32f50eaaf9a1 Author: Maxim Uvarov <maxim.uvarov(a)linaro.org> Date: Mon Sep 5 22:04:51 2016 +0300 example: l3fwd: fix clang build Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c index 4731237..082b2c2 100644 --- a/example/l3fwd/odp_l3fwd_db.c +++ b/example/l3fwd/odp_l3fwd_db.c @@ -209,7 +209,7 @@ static void create_fwd_hash_cache(void) } size = sizeof(flow_bucket_t) * bucket_count; - flows = (flow_entry_t *)((char *)bucket + size); + flows = (flow_entry_t *)(void *)((char *)bucket + size); fwd_lookup_cache.bucket = bucket; fwd_lookup_cache.bkt_cnt = bucket_count; ----------------------------------------------------------------------- Summary of changes: example/l3fwd/odp_l3fwd_db.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) hooks/post-receive --

10 years

[COMMIT] branch api-next updated. v1.10.1.0-175-g07bee0b

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, api-next has been updated via 07bee0be1f22f8a136982935ee06313e5bbc6f3a (commit) via 1594671cc63d7d1ac0192acb0a8496e7fec9f4a2 (commit) via b08633c5740311374f509cef5245f19bd53e55f7 (commit) via 6147511e8b2d8c5ae0a3fda161f346ba40203569 (commit) via ab82e67f3ea25b392c90557dfc4cad43750c8d94 (commit) via 6ec6cf6347efec4bf4268c6db37f287b9de35e30 (commit) via 442b93274d8f24f31557fe678245f65a2c499f97 (commit) via a4dd0bade1ca02eb91178834a159c9d26f140c38 (commit) via 4871852cc509ca998802616fadfbb5619e3324da (commit) via 48ae68961873bb818a75aa8aa1b93d697fe39517 (commit) via 85c4f0bd3f18900e484b0e71670fc18c555e75b1 (commit) via 57962f1b9c3a5a41a7dd6146f9666da18e62ec9c (commit) via 9a45802c94f58d9dd443c10e6dc204a58fa0f9ce (commit) from 0293146428ef9c6d8d2b24e0bdd6c781486904a5 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 07bee0be1f22f8a136982935ee06313e5bbc6f3a Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:46:03 2016 +0200 test: validation: drv: shmem: stress tests Stress tests, randomly allocating memory are added: the test is based on a group of odp threads allocating, mapping and freeing each-other memory. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.c b/test/common_plat/validation/drv/drvshmem/drvshmem.c index 9ca81fc..5e6d2e5 100644 --- a/test/common_plat/validation/drv/drvshmem/drvshmem.c +++ b/test/common_plat/validation/drv/drvshmem/drvshmem.c @@ -17,6 +17,25 @@ #define SMALL_MEM 10 #define MEDIUM_MEM 4096 #define BIG_MEM 16777216 +#define STRESS_SIZE 32 /* power of 2 and <=256 */ +#define STRESS_RANDOM_SZ 5 +#define STRESS_ITERATION 5000 + +typedef enum { + STRESS_FREE, /* entry is free and can be allocated */ + STRESS_BUSY, /* entry is being processed: don't touch */ + STRESS_ALLOC /* entry is allocated and can be freed */ +} stress_state_t; + +typedef struct { + stress_state_t state; + odpdrv_shm_t shm; + void *address; + uint32_t flags; + uint32_t size; + uint64_t align; + uint8_t data_val; +} stress_data_t; typedef struct { odpdrv_barrier_t test_barrier1; @@ -29,6 +48,8 @@ typedef struct { uint32_t nb_threads; odpdrv_shm_t shm[MAX_WORKERS]; void *address[MAX_WORKERS]; + odp_spinlock_t stress_lock; + stress_data_t stress[STRESS_SIZE]; } shared_test_data_t; /* memory stuff expected to fit in a single page */ @@ -543,10 +564,211 @@ void drvshmem_test_singleva_after_fork(void) CU_ASSERT(odpdrv_shm_print_all("Test completion") == base); } +/* + * thread part for the drvshmem_test_stress + */ +static int run_test_stress(void *arg ODP_UNUSED) +{ + odpdrv_shm_t shm; + uint8_t *address; + shared_test_data_t *glob_data; + uint8_t random_bytes[STRESS_RANDOM_SZ]; + uint32_t index; + uint32_t size; + uint64_t align; + uint32_t flags; + uint8_t data; + uint32_t iter; + uint32_t i; + + shm = odpdrv_shm_lookup_by_name(MEM_NAME); + glob_data = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(glob_data); + + /* wait for general GO! */ + odpdrv_barrier_wait(&glob_data->test_barrier1); + /* + + * at each iteration: pick up a random index for + * glob_data->stress[index]: If the entry is free, allocated mem + * randomly. If it is already allocated, make checks and free it: + * Note that different tread cann allocate or free a given block + */ + for (iter = 0; iter < STRESS_ITERATION; iter++) { + /* get 4 random bytes from which index, size ,align, flags + * and data will be derived: + */ + odp_random_data(random_bytes, STRESS_RANDOM_SZ, 0); + index = random_bytes[0] & (STRESS_SIZE - 1); + + odp_spinlock_lock(&glob_data->stress_lock); + + switch (glob_data->stress[index].state) { + case STRESS_FREE: + /* allocated a new block for this entry */ + + glob_data->stress[index].state = STRESS_BUSY; + odp_spinlock_unlock(&glob_data->stress_lock); + + size = (random_bytes[1] + 1) << 6; /* up to 16Kb */ + /* we just play with the VA flag. randomly setting + * the mlock flag may exceed user ulimit -l + */ + flags = random_bytes[2] & ODPDRV_SHM_SINGLE_VA; + align = (random_bytes[3] + 1) << 6;/* up to 16Kb */ + data = random_bytes[4]; + + shm = odpdrv_shm_reserve(NULL, size, align, flags); + glob_data->stress[index].shm = shm; + if (shm == ODPDRV_SHM_INVALID) { /* out of mem ? */ + odp_spinlock_lock(&glob_data->stress_lock); + glob_data->stress[index].state = STRESS_ALLOC; + odp_spinlock_unlock(&glob_data->stress_lock); + continue; + } + + address = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(address); + glob_data->stress[index].address = address; + glob_data->stress[index].flags = flags; + glob_data->stress[index].size = size; + glob_data->stress[index].align = align; + glob_data->stress[index].data_val = data; + + /* write some data: writing each byte would be a + * waste of time: just make sure each page is reached */ + for (i = 0; i < size; i += 256) + address[i] = (data++) & 0xFF; + odp_spinlock_lock(&glob_data->stress_lock); + glob_data->stress[index].state = STRESS_ALLOC; + odp_spinlock_unlock(&glob_data->stress_lock); + + break; + + case STRESS_ALLOC: + /* free the block for this entry */ + + glob_data->stress[index].state = STRESS_BUSY; + odp_spinlock_unlock(&glob_data->stress_lock); + shm = glob_data->stress[index].shm; + + if (shm == ODPDRV_SHM_INVALID) { /* out of mem ? */ + odp_spinlock_lock(&glob_data->stress_lock); + glob_data->stress[index].state = STRESS_FREE; + odp_spinlock_unlock(&glob_data->stress_lock); + continue; + } + + CU_ASSERT(odpdrv_shm_lookup_by_handle(shm) != 0); + + address = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(address); + + align = glob_data->stress[index].align; + if (align) { + align = glob_data->stress[index].align; + CU_ASSERT(((uintptr_t)address & (align - 1)) + == 0) + } + + flags = glob_data->stress[index].flags; + if (flags & ODPDRV_SHM_SINGLE_VA) + CU_ASSERT(glob_data->stress[index].address == + address) + + /* check that data is reachable and correct: */ + data = glob_data->stress[index].data_val; + size = glob_data->stress[index].size; + for (i = 0; i < size; i += 256) { + CU_ASSERT(address[i] == (data & 0xFF)); + data++; + } + + if (flags & ODPDRV_SHM_SINGLE_VA) { + CU_ASSERT(!odpdrv_shm_free_by_address(address)); + } else { + CU_ASSERT(!odpdrv_shm_free_by_handle(shm)); + } + + odp_spinlock_lock(&glob_data->stress_lock); + glob_data->stress[index].state = STRESS_FREE; + odp_spinlock_unlock(&glob_data->stress_lock); + + break; + + case STRESS_BUSY: + default: + odp_spinlock_unlock(&glob_data->stress_lock); + break; + } + } + + fflush(stdout); + return CU_get_number_of_failures(); +} + +/* + * stress tests + */ +void drvshmem_test_stress(void) +{ + pthrd_arg thrdarg; + odpdrv_shm_t shm; + shared_test_data_t *glob_data; + odp_cpumask_t unused; + int base; /* number of blocks already allocated at start of test */ + uint32_t i; + + base = odpdrv_shm_print_all("Before thread tests"); + + shm = odpdrv_shm_reserve(MEM_NAME, sizeof(shared_test_data_t), + 0, ODPDRV_SHM_LOCK); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + glob_data = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(glob_data); + + thrdarg.numthrds = odp_cpumask_default_worker(&unused, 0); + if (thrdarg.numthrds > MAX_WORKERS) + thrdarg.numthrds = MAX_WORKERS; + + glob_data->nb_threads = thrdarg.numthrds; + odpdrv_barrier_init(&glob_data->test_barrier1, thrdarg.numthrds); + odp_spinlock_init(&glob_data->stress_lock); + + /* before starting the threads, mark all entries as free: */ + for (i = 0; i < STRESS_SIZE; i++) + glob_data->stress[i].state = STRESS_FREE; + + /* create threads */ + odp_cunit_thread_create(run_test_stress, &thrdarg); + + /* wait for all thread endings: */ + CU_ASSERT(odp_cunit_thread_exit(&thrdarg) >= 0); + + odpdrv_shm_print_all("Middle"); + + /* release left overs: */ + for (i = 0; i < STRESS_SIZE; i++) { + shm = glob_data->stress[i].shm; + if ((glob_data->stress[i].state == STRESS_ALLOC) && + (glob_data->stress[i].shm != ODPDRV_SHM_INVALID)) { + CU_ASSERT(odpdrv_shm_lookup_by_handle(shm) != + NULL); + CU_ASSERT(!odpdrv_shm_free_by_handle(shm)); + } + } + + CU_ASSERT(0 == odpdrv_shm_free_by_name(MEM_NAME)); + + /* check that no memory is left over: */ + CU_ASSERT(odpdrv_shm_print_all("After stress tests") == base); +} + odp_testinfo_t drvshmem_suite[] = { ODP_TEST_INFO(drvshmem_test_basic), ODP_TEST_INFO(drvshmem_test_reserve_after_fork), ODP_TEST_INFO(drvshmem_test_singleva_after_fork), + ODP_TEST_INFO(drvshmem_test_stress), ODP_TEST_INFO_NULL, }; diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.h b/test/common_plat/validation/drv/drvshmem/drvshmem.h index 3f9f96e..f4c26a1 100644 --- a/test/common_plat/validation/drv/drvshmem/drvshmem.h +++ b/test/common_plat/validation/drv/drvshmem/drvshmem.h @@ -13,6 +13,7 @@ void drvshmem_test_basic(void); void drvshmem_test_reserve_after_fork(void); void drvshmem_test_singleva_after_fork(void); +void drvshmem_test_stress(void); /* test arrays: */ extern odp_testinfo_t drvshmem_suite[]; commit 1594671cc63d7d1ac0192acb0a8496e7fec9f4a2 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:46:02 2016 +0200 test: validation: drv: shmem: testing SINGLE_VA flag Basic test for different drvshm_reserve() time in conjunction with the usage of SINGLE_VA flag is added. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.c b/test/common_plat/validation/drv/drvshmem/drvshmem.c index 42ad844..9ca81fc 100644 --- a/test/common_plat/validation/drv/drvshmem/drvshmem.c +++ b/test/common_plat/validation/drv/drvshmem/drvshmem.c @@ -21,10 +21,14 @@ typedef struct { odpdrv_barrier_t test_barrier1; odpdrv_barrier_t test_barrier2; + odpdrv_barrier_t test_barrier3; + odpdrv_barrier_t test_barrier4; uint32_t foo; uint32_t bar; odpdrv_atomic_u32_t index; + uint32_t nb_threads; odpdrv_shm_t shm[MAX_WORKERS]; + void *address[MAX_WORKERS]; } shared_test_data_t; /* memory stuff expected to fit in a single page */ @@ -323,9 +327,226 @@ void drvshmem_test_reserve_after_fork(void) CU_ASSERT(odpdrv_shm_print_all("Test completion") == base); } +/* + * thread part for the drvshmem_test_singleva_after_fork + */ +static int run_test_singleva_after_fork(void *arg ODP_UNUSED) +{ + odpdrv_shm_t shm; + shared_test_data_t *glob_data; + int thr; + int thr_index; + char *name; + int name_len; + int size; + shared_test_data_small_t *pattern_small; + shared_test_data_medium_t *pattern_medium; + shared_test_data_big_t *pattern_big; + uint32_t i; + int ret; + + thr = odp_thread_id(); + printf("Thread %i starts\n", thr); + + shm = odpdrv_shm_lookup_by_name(MEM_NAME); + glob_data = odpdrv_shm_addr(shm); + + /* + * odp_thread_id are not guaranteed to be consecutive, so we create + * a consecutive ID + */ + thr_index = odpdrv_atomic_fetch_inc_u32(&glob_data->index); + + /* allocate some memory (of different sizes) and fill with pattern */ + name_len = strlen(MEM_NAME) + 20; + name = malloc(name_len); + snprintf(name, name_len, "%s-%09d", MEM_NAME, thr_index); + switch (thr_index % 3) { + case 0: + size = sizeof(shared_test_data_small_t); + shm = odpdrv_shm_reserve(name, size, 0, ODPDRV_SHM_SINGLE_VA); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + glob_data->shm[thr_index] = shm; + pattern_small = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(pattern_small); + glob_data->address[thr_index] = (void *)pattern_small; + for (i = 0; i < SMALL_MEM; i++) + pattern_small->data[i] = i; + break; + case 1: + size = sizeof(shared_test_data_medium_t); + shm = odpdrv_shm_reserve(name, size, 0, ODPDRV_SHM_SINGLE_VA); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + glob_data->shm[thr_index] = shm; + pattern_medium = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(pattern_medium); + glob_data->address[thr_index] = (void *)pattern_medium; + for (i = 0; i < MEDIUM_MEM; i++) + pattern_medium->data[i] = (i << 2); + break; + case 2: + size = sizeof(shared_test_data_big_t); + shm = odpdrv_shm_reserve(name, size, 0, ODPDRV_SHM_SINGLE_VA); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + glob_data->shm[thr_index] = shm; + pattern_big = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(pattern_big); + glob_data->address[thr_index] = (void *)pattern_big; + for (i = 0; i < BIG_MEM; i++) + pattern_big->data[i] = (i >> 2); + break; + } + free(name); + + /* print block address */ + printf("In thread: Block index: %d mapped at %lx\n", + thr_index, (long int)odpdrv_shm_addr(shm)); + + odpdrv_barrier_wait(&glob_data->test_barrier1); + odpdrv_barrier_wait(&glob_data->test_barrier2); + + /* map each-other block, checking common address: */ + for (i = 0; i < glob_data->nb_threads; i++) { + shm = odpdrv_shm_lookup_by_address(glob_data->address[i]); + CU_ASSERT(shm == glob_data->shm[i]); + CU_ASSERT(odpdrv_shm_addr(shm) == glob_data->address[i]); + } + + /* wait for main control task and free the allocated block */ + odpdrv_barrier_wait(&glob_data->test_barrier3); + odpdrv_barrier_wait(&glob_data->test_barrier4); + ret = odpdrv_shm_free_by_address(glob_data->address[thr_index]); + CU_ASSERT(ret == 0); + + fflush(stdout); + return CU_get_number_of_failures(); +} + +/* + * test sharing memory reserved after odp_thread creation (e.g. fork()): + * with single VA flag. + */ +void drvshmem_test_singleva_after_fork(void) +{ + int base; /* memory usage when test starts */ + pthrd_arg thrdarg; + odpdrv_shm_t shm; + odpdrv_shm_t thr_shm; + shared_test_data_t *glob_data; + odp_cpumask_t unused; + char *name; + int name_len; + int thr_index; + int i; + void *address; + shared_test_data_small_t *pattern_small; + shared_test_data_medium_t *pattern_medium; + shared_test_data_big_t *pattern_big; + + base = odpdrv_shm_print_all("Before drvshmem_test_singleva_after_fork"); + + shm = odpdrv_shm_reserve(MEM_NAME, sizeof(shared_test_data_t), + 0, ODPDRV_SHM_LOCK); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + glob_data = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(glob_data); + + thrdarg.numthrds = odp_cpumask_default_worker(&unused, 0); + if (thrdarg.numthrds > MAX_WORKERS) + thrdarg.numthrds = MAX_WORKERS; + + glob_data->nb_threads = thrdarg.numthrds; + odpdrv_barrier_init(&glob_data->test_barrier1, thrdarg.numthrds + 1); + odpdrv_barrier_init(&glob_data->test_barrier2, thrdarg.numthrds + 1); + odpdrv_barrier_init(&glob_data->test_barrier3, thrdarg.numthrds + 1); + odpdrv_barrier_init(&glob_data->test_barrier4, thrdarg.numthrds + 1); + odpdrv_atomic_store_u32(&glob_data->index, 0); + + odp_cunit_thread_create(run_test_singleva_after_fork, &thrdarg); + + /* wait until all threads have made their shm_reserve: */ + odpdrv_barrier_wait(&glob_data->test_barrier1); + CU_ASSERT(odpdrv_shm_print_all("After thread reserve") + == base + thrdarg.numthrds + 1); + + /* perform a lookup of all memories, by handle or name: */ + name_len = strlen(MEM_NAME) + 20; + name = malloc(name_len); + for (thr_index = 0; thr_index < thrdarg.numthrds; thr_index++) { + if (thr_index % 2) { + snprintf(name, name_len, "%s-%09d", + MEM_NAME, thr_index); + thr_shm = odpdrv_shm_lookup_by_name(name); + CU_ASSERT(thr_shm == glob_data->shm[thr_index]); + } else { + odpdrv_shm_lookup_by_handle(glob_data->shm[thr_index]); + } + } + free(name); + + /* check that the patterns are correct: */ + for (thr_index = 0; thr_index < thrdarg.numthrds; thr_index++) { + switch (thr_index % 3) { + case 0: + pattern_small = + odpdrv_shm_addr(glob_data->shm[thr_index]); + CU_ASSERT_PTR_NOT_NULL(pattern_small); + for (i = 0; i < SMALL_MEM; i++) + CU_ASSERT(pattern_small->data[i] == i); + break; + case 1: + pattern_medium = + odpdrv_shm_addr(glob_data->shm[thr_index]); + CU_ASSERT_PTR_NOT_NULL(pattern_medium); + for (i = 0; i < MEDIUM_MEM; i++) + CU_ASSERT(pattern_medium->data[i] == (i << 2)); + break; + case 2: + pattern_big = + odpdrv_shm_addr(glob_data->shm[thr_index]); + CU_ASSERT_PTR_NOT_NULL(pattern_big); + for (i = 0; i < BIG_MEM; i++) + CU_ASSERT(pattern_big->data[i] == (i >> 2)); + break; + } + } + + /* + * check that the mapping address is common to all (SINGLE_VA): + */ + for (thr_index = 0; thr_index < thrdarg.numthrds; thr_index++) { + address = odpdrv_shm_addr(glob_data->shm[thr_index]); + CU_ASSERT(glob_data->address[thr_index] == address); + } + + CU_ASSERT(odpdrv_shm_print_all("After local lookup") + == base + thrdarg.numthrds + 1); + + /* unblock the threads and let them map each-other blocks: */ + odpdrv_barrier_wait(&glob_data->test_barrier2); + + /* then check mem status */ + odpdrv_barrier_wait(&glob_data->test_barrier3); + CU_ASSERT(odpdrv_shm_print_all("After mutual lookup") + == base + thrdarg.numthrds + 1); + + /* unblock the threads and let them free all thread blocks: */ + odpdrv_barrier_wait(&glob_data->test_barrier4); + + /* wait for all thread endings: */ + CU_ASSERT(odp_cunit_thread_exit(&thrdarg) >= 0); + + /* just glob_data should remain: */ + CU_ASSERT(odpdrv_shm_print_all("After threads free") == base + 1); + + CU_ASSERT(0 == odpdrv_shm_free_by_name(MEM_NAME)); + CU_ASSERT(odpdrv_shm_print_all("Test completion") == base); +} + odp_testinfo_t drvshmem_suite[] = { ODP_TEST_INFO(drvshmem_test_basic), ODP_TEST_INFO(drvshmem_test_reserve_after_fork), + ODP_TEST_INFO(drvshmem_test_singleva_after_fork), ODP_TEST_INFO_NULL, }; diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.h b/test/common_plat/validation/drv/drvshmem/drvshmem.h index 12d738e..3f9f96e 100644 --- a/test/common_plat/validation/drv/drvshmem/drvshmem.h +++ b/test/common_plat/validation/drv/drvshmem/drvshmem.h @@ -12,6 +12,7 @@ /* test functions: */ void drvshmem_test_basic(void); void drvshmem_test_reserve_after_fork(void); +void drvshmem_test_singleva_after_fork(void); /* test arrays: */ extern odp_testinfo_t drvshmem_suite[]; commit b08633c5740311374f509cef5245f19bd53e55f7 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:46:01 2016 +0200 test: validation: drv: shmem: basic tests Adding simple tests for shmem operations performed on the driver (drv) interface. Does not to assume pthread as odpthreads (the barrier is now part of the reserved area, rather than a global which only works for pthreads). Tests simple functionanlity, but regardless of reserve() time (i.e. memory reservation occurs both before and after odpthread creation here). Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/m4/configure.m4 b/test/common_plat/m4/configure.m4 index 20c3a3b..1fc350d 100644 --- a/test/common_plat/m4/configure.m4 +++ b/test/common_plat/m4/configure.m4 @@ -32,4 +32,5 @@ AC_CONFIG_FILES([test/common_plat/Makefile test/common_plat/validation/api/timer/Makefile test/common_plat/validation/api/traffic_mngr/Makefile test/common_plat/validation/drv/Makefile - test/common_plat/validation/drv/drvatomic/Makefile]) + test/common_plat/validation/drv/drvatomic/Makefile + test/common_plat/validation/drv/drvshmem/Makefile]) diff --git a/test/common_plat/validation/drv/Makefile.am b/test/common_plat/validation/drv/Makefile.am index eedbad5..bcdb92e 100644 --- a/test/common_plat/validation/drv/Makefile.am +++ b/test/common_plat/validation/drv/Makefile.am @@ -1,4 +1,5 @@ -ODPDRV_MODULES = drvatomic +ODPDRV_MODULES = drvatomic \ + drvshmem SUBDIRS = $(ODPDRV_MODULES) diff --git a/test/common_plat/validation/drv/drvshmem/.gitignore b/test/common_plat/validation/drv/drvshmem/.gitignore new file mode 100644 index 0000000..726dea3 --- /dev/null +++ b/test/common_plat/validation/drv/drvshmem/.gitignore @@ -0,0 +1 @@ +drvshmem_main diff --git a/test/common_plat/validation/drv/drvshmem/Makefile.am b/test/common_plat/validation/drv/drvshmem/Makefile.am new file mode 100644 index 0000000..816a5e6 --- /dev/null +++ b/test/common_plat/validation/drv/drvshmem/Makefile.am @@ -0,0 +1,10 @@ +include ../Makefile.inc + +noinst_LTLIBRARIES = libtestdrvshmem.la +libtestdrvshmem_la_SOURCES = drvshmem.c + +test_PROGRAMS = drvshmem_main$(EXEEXT) +dist_drvshmem_main_SOURCES = drvshmem_main.c +drvshmem_main_LDADD = libtestdrvshmem.la $(LIBCUNIT_COMMON) $(LIBODP) + +EXTRA_DIST = drvshmem.h diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.c b/test/common_plat/validation/drv/drvshmem/drvshmem.c new file mode 100644 index 0000000..42ad844 --- /dev/null +++ b/test/common_plat/validation/drv/drvshmem/drvshmem.c @@ -0,0 +1,351 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp_drv.h> +#include <odp_api.h> +#include <odp_cunit_common.h> +#include "drvshmem.h" +#include <stdlib.h> + +#define ALIGN_SIZE (128) +#define MEM_NAME "test_shmem" +#define TEST_SHARE_FOO (0xf0f0f0f0) +#define TEST_SHARE_BAR (0xf0f0f0f) +#define SMALL_MEM 10 +#define MEDIUM_MEM 4096 +#define BIG_MEM 16777216 + +typedef struct { + odpdrv_barrier_t test_barrier1; + odpdrv_barrier_t test_barrier2; + uint32_t foo; + uint32_t bar; + odpdrv_atomic_u32_t index; + odpdrv_shm_t shm[MAX_WORKERS]; +} shared_test_data_t; + +/* memory stuff expected to fit in a single page */ +typedef struct { + int data[SMALL_MEM]; +} shared_test_data_small_t; + +/* memory stuff expected to fit in a huge page */ +typedef struct { + int data[MEDIUM_MEM]; +} shared_test_data_medium_t; + +/* memory stuff expected to fit in many huge pages */ +typedef struct { + int data[BIG_MEM]; +} shared_test_data_big_t; + +/* + * thread part for the drvshmem_test_basic test + */ +static int run_test_basic_thread(void *arg ODP_UNUSED) +{ + odpdrv_shm_info_t info; + odpdrv_shm_t shm; + shared_test_data_t *shared_test_data; + int thr; + + thr = odp_thread_id(); + printf("Thread %i starts\n", thr); + + shm = odpdrv_shm_lookup_by_name(MEM_NAME); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + shared_test_data = odpdrv_shm_addr(shm); + CU_ASSERT(NULL != shared_test_data); + + odpdrv_barrier_wait(&shared_test_data->test_barrier1); + odpdrv_shm_print_all("(from thread) After lookup for the global shmem"); + CU_ASSERT(TEST_SHARE_FOO == shared_test_data->foo); + CU_ASSERT(TEST_SHARE_BAR == shared_test_data->bar); + CU_ASSERT(0 == odpdrv_shm_info(shm, &info)); + CU_ASSERT(0 == strcmp(MEM_NAME, info.name)); + CU_ASSERT(0 == info.flags); + CU_ASSERT(shared_test_data == info.addr); + CU_ASSERT(sizeof(shared_test_data_t) <= info.size); +#ifdef MAP_HUGETLB + CU_ASSERT(odp_sys_huge_page_size() == info.page_size); +#else + CU_ASSERT(odp_sys_page_size() == info.page_size); +#endif + odpdrv_shm_print_all("(from thread) About to end"); + + fflush(stdout); + return CU_get_number_of_failures(); +} + +/* + * test basic things: shmem creation, info, share, and free + */ +void drvshmem_test_basic(void) +{ + int base; /* memory usage when test starts */ + pthrd_arg thrdarg; + odpdrv_shm_t shm; + shared_test_data_t *shared_test_data; + odp_cpumask_t unused; + + base = odpdrv_shm_print_all("Before drvshmem_test_basic"); + shm = odpdrv_shm_reserve(MEM_NAME, + sizeof(shared_test_data_t), ALIGN_SIZE, 0); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + CU_ASSERT(odpdrv_shm_to_u64(shm) != + odpdrv_shm_to_u64(ODPDRV_SHM_INVALID)); + + CU_ASSERT(0 == odpdrv_shm_free_by_handle(shm)); + CU_ASSERT(ODPDRV_SHM_INVALID == odpdrv_shm_lookup_by_name(MEM_NAME)); + + shm = odpdrv_shm_reserve(MEM_NAME, + sizeof(shared_test_data_t), ALIGN_SIZE, 0); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + + shared_test_data = odpdrv_shm_addr(shm); + CU_ASSERT_FATAL(NULL != shared_test_data); + shared_test_data->foo = TEST_SHARE_FOO; + shared_test_data->bar = TEST_SHARE_BAR; + + CU_ASSERT(odpdrv_shm_free_by_address((char *)shared_test_data - 1) < 0); + + thrdarg.numthrds = odp_cpumask_default_worker(&unused, 0); + + if (thrdarg.numthrds > MAX_WORKERS) + thrdarg.numthrds = MAX_WORKERS; + + odpdrv_barrier_init(&shared_test_data->test_barrier1, thrdarg.numthrds); + odp_cunit_thread_create(run_test_basic_thread, &thrdarg); + CU_ASSERT(odp_cunit_thread_exit(&thrdarg) >= 0); + + CU_ASSERT(0 == odpdrv_shm_free_by_handle(shm)); + CU_ASSERT(odpdrv_shm_print_all("Test completion") == base); +} + +/* + * thread part for the drvshmem_test_reserve_after_fork + */ +static int run_test_reserve_after_fork(void *arg ODP_UNUSED) +{ + odpdrv_shm_t shm; + shared_test_data_t *glob_data; + int thr; + int thr_index; + char *name; + int name_len; + int size; + shared_test_data_small_t *pattern_small; + shared_test_data_medium_t *pattern_medium; + shared_test_data_big_t *pattern_big; + int i; + + thr = odp_thread_id(); + printf("Thread %i starts\n", thr); + + shm = odpdrv_shm_lookup_by_name(MEM_NAME); + glob_data = odpdrv_shm_addr(shm); + + /* + * odp_thread_id are not guaranteed to be consecutive, so we create + * a consecutive ID + */ + thr_index = odpdrv_atomic_fetch_inc_u32(&glob_data->index); + + /* allocate some memory (of different sizes) and fill with pattern */ + name_len = strlen(MEM_NAME) + 20; + name = malloc(name_len); + snprintf(name, name_len, "%s-%09d", MEM_NAME, thr_index); + switch (thr_index % 3) { + case 0: + size = sizeof(shared_test_data_small_t); + shm = odpdrv_shm_reserve(name, size, 0, 0); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + glob_data->shm[thr_index] = shm; + pattern_small = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(pattern_small); + for (i = 0; i < SMALL_MEM; i++) + pattern_small->data[i] = i; + break; + case 1: + size = sizeof(shared_test_data_medium_t); + shm = odpdrv_shm_reserve(name, size, 0, 0); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + glob_data->shm[thr_index] = shm; + pattern_medium = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(pattern_medium); + for (i = 0; i < MEDIUM_MEM; i++) + pattern_medium->data[i] = (i << 2); + break; + case 2: + size = sizeof(shared_test_data_big_t); + shm = odpdrv_shm_reserve(name, size, 0, 0); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + glob_data->shm[thr_index] = shm; + pattern_big = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(pattern_big); + for (i = 0; i < BIG_MEM; i++) + pattern_big->data[i] = (i >> 2); + break; + } + free(name); + + /* print block address */ + printf("In thread: Block index: %d mapped at %lx\n", + thr_index, (long int)odpdrv_shm_addr(shm)); + + odpdrv_barrier_wait(&glob_data->test_barrier1); + odpdrv_barrier_wait(&glob_data->test_barrier2); + + fflush(stdout); + return CU_get_number_of_failures(); +} + +/* + * test sharing memory reserved after odp_thread creation (e.g. fork()): + */ +void drvshmem_test_reserve_after_fork(void) +{ + int base; /* memory usage when test starts */ + pthrd_arg thrdarg; + odpdrv_shm_t shm; + odpdrv_shm_t thr_shm; + shared_test_data_t *glob_data; + odp_cpumask_t unused; + char *name; + int name_len; + int thr_index; + int i; + void *address; + shared_test_data_small_t *pattern_small; + shared_test_data_medium_t *pattern_medium; + shared_test_data_big_t *pattern_big; + + base = odpdrv_shm_print_all("Before drvshmem_test_reserve_after_fork"); + shm = odpdrv_shm_reserve(MEM_NAME, sizeof(shared_test_data_t), 0, 0); + CU_ASSERT(ODPDRV_SHM_INVALID != shm); + glob_data = odpdrv_shm_addr(shm); + CU_ASSERT_PTR_NOT_NULL(glob_data); + + thrdarg.numthrds = odp_cpumask_default_worker(&unused, 0); + if (thrdarg.numthrds > MAX_WORKERS) + thrdarg.numthrds = MAX_WORKERS; + + odpdrv_barrier_init(&glob_data->test_barrier1, thrdarg.numthrds + 1); + odpdrv_barrier_init(&glob_data->test_barrier2, thrdarg.numthrds + 1); + odpdrv_atomic_store_u32(&glob_data->index, 0); + + odp_cunit_thread_create(run_test_reserve_after_fork, &thrdarg); + + /* wait until all threads have made their shm_reserve: */ + odpdrv_barrier_wait(&glob_data->test_barrier1); + CU_ASSERT(odpdrv_shm_print_all("After all thread reserve") + == base + thrdarg.numthrds + 1); + + /* perform a lookup of all memories, by handle or name: */ + name_len = strlen(MEM_NAME) + 20; + name = malloc(name_len); + for (thr_index = 0; thr_index < thrdarg.numthrds; thr_index++) { + if (thr_index % 2) { + snprintf(name, name_len, "%s-%09d", + MEM_NAME, thr_index); + thr_shm = odpdrv_shm_lookup_by_name(name); + CU_ASSERT(thr_shm == glob_data->shm[thr_index]); + } else { + odpdrv_shm_lookup_by_handle(glob_data->shm[thr_index]); + } + } + + /* check that the patterns are correct: */ + for (thr_index = 0; thr_index < thrdarg.numthrds; thr_index++) { + switch (thr_index % 3) { + case 0: + pattern_small = + odpdrv_shm_addr(glob_data->shm[thr_index]); + CU_ASSERT_PTR_NOT_NULL(pattern_small); + for (i = 0; i < SMALL_MEM; i++) + CU_ASSERT(pattern_small->data[i] == i); + break; + case 1: + pattern_medium = + odpdrv_shm_addr(glob_data->shm[thr_index]); + CU_ASSERT_PTR_NOT_NULL(pattern_medium); + for (i = 0; i < MEDIUM_MEM; i++) + CU_ASSERT(pattern_medium->data[i] == (i << 2)); + break; + case 2: + pattern_big = + odpdrv_shm_addr(glob_data->shm[thr_index]); + CU_ASSERT_PTR_NOT_NULL(pattern_big); + for (i = 0; i < BIG_MEM; i++) + CU_ASSERT(pattern_big->data[i] == (i >> 2)); + break; + } + } + + /* + * print the mapping address of the blocks + */ + for (thr_index = 0; thr_index < thrdarg.numthrds; thr_index++) { + address = odpdrv_shm_addr(glob_data->shm[thr_index]); + printf("In main Block index: %d mapped at %lx\n", + thr_index, (long int)address); + } + + CU_ASSERT(odpdrv_shm_print_all("After main lookup of thread shmem") + == base + thrdarg.numthrds + 1); + + /* unblock the threads and let them terminate (no free is done): */ + odpdrv_barrier_wait(&glob_data->test_barrier2); + + /* at the same time, (race),free of all memories, by handle or name: */ + for (thr_index = 0; thr_index < thrdarg.numthrds; thr_index++) { + if (thr_index % 2) { + thr_shm = glob_data->shm[thr_index]; + CU_ASSERT(odpdrv_shm_free_by_handle(thr_shm) == 0); + } else { + snprintf(name, name_len, "%s-%09d", + MEM_NAME, thr_index); + CU_ASSERT(odpdrv_shm_free_by_name(name) == 0); + } + } + free(name); + + /* wait for all thread endings: */ + CU_ASSERT(odp_cunit_thread_exit(&thrdarg) >= 0); + + /* just glob_data should remain: */ + CU_ASSERT(odpdrv_shm_print_all("After all threads end") == base + 1); + + CU_ASSERT(0 == odpdrv_shm_free_by_handle(shm)); + CU_ASSERT(odpdrv_shm_print_all("Test completion") == base); +} + +odp_testinfo_t drvshmem_suite[] = { + ODP_TEST_INFO(drvshmem_test_basic), + ODP_TEST_INFO(drvshmem_test_reserve_after_fork), + ODP_TEST_INFO_NULL, +}; + +odp_suiteinfo_t drvshmem_suites[] = { + {"Shared Memory", NULL, NULL, drvshmem_suite}, + ODP_SUITE_INFO_NULL, +}; + +int drvshmem_main(int argc, char *argv[]) +{ + int ret; + + /* parse common options: */ + if (odp_cunit_parse_options(argc, argv)) + return -1; + + ret = odp_cunit_register(drvshmem_suites); + + if (ret == 0) + ret = odp_cunit_run(); + + return ret; +} diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.h b/test/common_plat/validation/drv/drvshmem/drvshmem.h new file mode 100644 index 0000000..12d738e --- /dev/null +++ b/test/common_plat/validation/drv/drvshmem/drvshmem.h @@ -0,0 +1,25 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ODP_TEST_DRVSHMEM_H_ +#define _ODP_TEST_DRVSHMEM_H_ + +#include <odp_cunit_common.h> + +/* test functions: */ +void drvshmem_test_basic(void); +void drvshmem_test_reserve_after_fork(void); + +/* test arrays: */ +extern odp_testinfo_t drvshmem_suite[]; + +/* test registry: */ +extern odp_suiteinfo_t drvshmem_suites[]; + +/* main test program: */ +int drvshmem_main(int argc, char *argv[]); + +#endif diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem_main.c b/test/common_plat/validation/drv/drvshmem/drvshmem_main.c new file mode 100644 index 0000000..566ccec --- /dev/null +++ b/test/common_plat/validation/drv/drvshmem/drvshmem_main.c @@ -0,0 +1,12 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include "drvshmem.h" + +int main(int argc, char *argv[]) +{ + return drvshmem_main(argc, argv); +} diff --git a/test/linux-generic/Makefile.am b/test/linux-generic/Makefile.am index 23b8e03..2b7cbc3 100644 --- a/test/linux-generic/Makefile.am +++ b/test/linux-generic/Makefile.am @@ -35,6 +35,7 @@ TESTS = validation/api/pktio/pktio_run.sh \ $(ALL_API_VALIDATION_DIR)/shmem/shmem_main$(EXEEXT) \ $(ALL_API_VALIDATION_DIR)/system/system_main$(EXEEXT) \ $(ALL_DRV_VALIDATION_DIR)/drvatomic/drvatomic_main$(EXEEXT) \ + $(ALL_DRV_VALIDATION_DIR)/drvshmem/drvshmem_main$(EXEEXT) \ ring/ring_main$(EXEEXT) SUBDIRS += validation/api/pktio\ commit 6147511e8b2d8c5ae0a3fda161f346ba40203569 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:46:00 2016 +0200 linux-gen: drv: shm The shared memory DRV interface is created, mostly as a wrapper around the internal memory allocator, ishm. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/include/odp_drv.h b/include/odp_drv.h index 31c620e..0959879 100644 --- a/include/odp_drv.h +++ b/include/odp_drv.h @@ -23,6 +23,7 @@ extern C { #include <odp/drv/barrier.h> #include <odp/drv/byteorder.h> #include <odp/drv/compiler.h> +#include <odp/drv/shm.h> #include <odp/drv/spinlock.h> #include <odp/drv/std_types.h> #include <odp/drv/sync.h> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index ac2b2da..4d9b84b 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -99,6 +99,7 @@ odpdrvinclude_HEADERS = \ $(srcdir)/include/odp/drv/barrier.h \ $(srcdir)/include/odp/drv/byteorder.h \ $(srcdir)/include/odp/drv/compiler.h \ + $(srcdir)/include/odp/drv/shm.h \ $(srcdir)/include/odp/drv/spinlock.h \ $(srcdir)/include/odp/drv/std_types.h \ $(srcdir)/include/odp/drv/sync.h @@ -108,6 +109,7 @@ odpdrvplatinclude_HEADERS = \ $(srcdir)/include/odp/drv/plat/atomic_types.h \ $(srcdir)/include/odp/drv/plat/barrier_types.h \ $(srcdir)/include/odp/drv/plat/byteorder_types.h \ + $(srcdir)/include/odp/drv/plat/shm_types.h \ $(srcdir)/include/odp/drv/plat/spinlock_types.h \ $(srcdir)/include/odp/drv/plat/strong_types.h @@ -214,6 +216,7 @@ __LIB__libodp_linux_la_SOURCES = \ odp_weak.c \ drv_atomic.c \ drv_barrier.c \ + drv_shm.c \ drv_spinlock.c \ arch/@ARCH_DIR@/odp_cpu_arch.c \ arch/@ARCH_DIR@/odp_sysinfo_parse.c diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index c79c58e..f4aa6d3 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -52,6 +52,7 @@ #include <odp/api/align.h> #include <odp/api/system_info.h> #include <odp/api/debug.h> +#include <odp/drv/shm.h> #include <odp_shm_internal.h> #include <odp_debug_internal.h> #include <odp_align_internal.h> @@ -79,13 +80,12 @@ * if some of the block ownwers never procsync() after free). This number * should take that into account) */ -#define ISHM_MAX_NB_BLOCKS 128 +#define ISHM_MAX_NB_BLOCKS ODPDRV_CONFIG_SHM_BLOCKS /* * Maximum internal shared memory block name length in chars - * probably taking the same number as SHM name size make sense at this stage */ -#define ISHM_NAME_MAXLEN 32 +#define ISHM_NAME_MAXLEN ODPDRV_SHM_NAME_LEN /* * Linux underlying file name: <directory>/odp-<odp_pid>-ishm-<name> diff --git a/platform/linux-generic/drv_shm.c b/platform/linux-generic/drv_shm.c new file mode 100644 index 0000000..9b2560d --- /dev/null +++ b/platform/linux-generic/drv_shm.c @@ -0,0 +1,102 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp_config_internal.h> +#include <odp/api/std_types.h> +#include <odp/drv/shm.h> +#include <_ishm_internal.h> + +static inline uint32_t from_handle(odpdrv_shm_t shm) +{ + return _odpdrv_typeval(shm) - 1; +} + +static inline odpdrv_shm_t to_handle(uint32_t index) +{ + return _odpdrv_cast_scalar(odpdrv_shm_t, index + 1); +} + +int odpdrv_shm_capability(odpdrv_shm_capability_t *capa) +{ + capa->max_blocks = ODPDRV_CONFIG_SHM_BLOCKS; + capa->max_size = 0; + capa->max_align = 0; + + return 0; +} + +odpdrv_shm_t odpdrv_shm_reserve(const char *name, uint64_t size, uint64_t align, + uint32_t flags) +{ + int block_index; + int flgs = 0; /* internal ishm flags */ + + /* set internal ishm flags according to API flags: */ + flgs |= (flags & ODPDRV_SHM_SINGLE_VA) ? _ODP_ISHM_SINGLE_VA : 0; + flgs |= (flags & ODPDRV_SHM_LOCK) ? _ODP_ISHM_LOCK : 0; + + block_index = _odp_ishm_reserve(name, size, -1, align, flgs, flags); + if (block_index >= 0) + return to_handle(block_index); + else + return ODPDRV_SHM_INVALID; +} + +int odpdrv_shm_free_by_handle(odpdrv_shm_t shm) +{ + return _odp_ishm_free_by_index(from_handle(shm)); +} + +int odpdrv_shm_free_by_name(const char *name) +{ + return _odp_ishm_free_by_name(name); +} + +int odpdrv_shm_free_by_address(void *address) +{ + return _odp_ishm_free_by_address(address); +} + +void *odpdrv_shm_lookup_by_handle(odpdrv_shm_t shm) +{ + return _odp_ishm_lookup_by_index(from_handle(shm)); +} + +odpdrv_shm_t odpdrv_shm_lookup_by_name(const char *name) +{ + return to_handle(_odp_ishm_lookup_by_name(name)); +} + +odpdrv_shm_t odpdrv_shm_lookup_by_address(void *address) +{ + return to_handle(_odp_ishm_lookup_by_address(address)); +} + +void *odpdrv_shm_addr(odpdrv_shm_t shm) +{ + return _odp_ishm_address(from_handle(shm)); +} + +int odpdrv_shm_info(odpdrv_shm_t shm, odpdrv_shm_info_t *info) +{ + _odp_ishm_info_t ishm_info; + + if (_odp_ishm_info(from_handle(shm), &ishm_info)) + return -1; + + info->name = ishm_info.name; + info->addr = ishm_info.addr; + info->size = ishm_info.size; + info->page_size = ishm_info.page_size; + info->flags = ishm_info.user_flags; + + return 0; +} + +int odpdrv_shm_print_all(const char *title) +{ + return _odp_ishm_status(title); +} diff --git a/platform/linux-generic/include/odp/drv/plat/shm_types.h b/platform/linux-generic/include/odp/drv/plat/shm_types.h new file mode 100644 index 0000000..c48eeca --- /dev/null +++ b/platform/linux-generic/include/odp/drv/plat/shm_types.h @@ -0,0 +1,46 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODP shared memory + */ + +#ifndef ODPDRV_SHM_TYPES_H_ +#define ODPDRV_SHM_TYPES_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp/drv/std_types.h> +#include <odp/drv/plat/strong_types.h> + +/** @addtogroup odpdrv_shm ODPDRV SHARED MEMORY + * Operations on driver shared memory. + * @{ + */ + +typedef ODPDRV_HANDLE_T(odpdrv_shm_t); + +#define ODPDRV_SHM_INVALID _odpdrv_cast_scalar(odpdrv_shm_t, 0) + +/** Get printable format of odpdrv_shm_t */ +static inline uint64_t odpdrv_shm_to_u64(odpdrv_shm_t hdl) +{ + return _odpdrv_pri(hdl); +} + +/** + * @} + */ + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp/drv/shm.h b/platform/linux-generic/include/odp/drv/shm.h new file mode 100644 index 0000000..644b036 --- /dev/null +++ b/platform/linux-generic/include/odp/drv/shm.h @@ -0,0 +1,36 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODPDRV shared memory + */ + +#ifndef ODPDRV_PLAT_SHM_H_ +#define ODPDRV_PLAT_SHM_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp/drv/plat/shm_types.h> + +/** @ingroup odpdrv_shm + * @{ + */ + +/** + * @} + */ + +#include <odp/drv/spec/shm.h> + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp_config_internal.h b/platform/linux-generic/include/odp_config_internal.h index 3eabc7b..7d5dbeb 100644 --- a/platform/linux-generic/include/odp_config_internal.h +++ b/platform/linux-generic/include/odp_config_internal.h @@ -120,6 +120,12 @@ extern "C" { */ #define ODP_CONFIG_ISHM_VA_PREALLOC_SZ (536870912L) +/* Maximum number of shared memory blocks available on the driver interface. + * + * This the the number of separate SHM areas that can be reserved concurrently + */ +#define ODPDRV_CONFIG_SHM_BLOCKS 48 + #ifdef __cplusplus } #endif commit ab82e67f3ea25b392c90557dfc4cad43750c8d94 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:45:59 2016 +0200 linux-gen: ishm: adding debug function A debug function, printing the internal mem alloc status is added. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index c8156aa..c79c58e 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -1333,3 +1333,128 @@ int _odp_ishm_term_local(void) odp_spinlock_unlock(&ishm_tbl->lock); return 0; } + +/* + * Print the current ishm status (allocated blocks and VA space map) + * Return the number of allocated blocks (including those not mapped + * by the current odp thread). Also perform a number of sanity check. + * For debug. + */ +int _odp_ishm_status(const char *title) +{ + int i; + char flags[3]; + char huge; + int proc_index; + ishm_fragment_t *fragmnt; + int consecutive_unallocated = 0; /* should never exceed 1 */ + uintptr_t last_address = 0; + ishm_fragment_t *previous = NULL; + int nb_used_frgments = 0; + int nb_unused_frgments = 0; /* nb frag describing a VA area */ + int nb_allocated_frgments = 0; /* nb frag describing an allocated VA */ + int nb_blocks = 0; + int single_va_blocks = 0; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + ODP_DBG("ishm blocks allocated at: %s\n", title); + + /* display block table: 1 line per entry +1 extra line if mapped here */ + for (i = 0; i < ISHM_MAX_NB_BLOCKS; i++) { + if (ishm_tbl->block[i].len <= 0) + continue; /* unused block */ + + nb_blocks++; + if (ishm_tbl->block[i].flags & _ODP_ISHM_SINGLE_VA) + single_va_blocks++; + + flags[0] = (ishm_tbl->block[i].flags & _ODP_ISHM_SINGLE_VA) ? + 'S' : '.'; + flags[1] = (ishm_tbl->block[i].flags & _ODP_ISHM_LOCK) ? + 'L' : '.'; + flags[2] = 0; + huge = (ishm_tbl->block[i].huge) ? 'H' : '.'; + proc_index = procfind_block(i); + ODP_DBG("%-3d: name:%-.24s file:%-.24s tid:%-3d" + " flags:%s,%c len:0x%-08lx" + " user_len:%-8ld seq:%-3ld refcnt:%-4d\n", + i, + ishm_tbl->block[i].name, + ishm_tbl->block[i].filename, + ishm_tbl->block[i].main_odpthread, + flags, huge, + ishm_tbl->block[i].len, + ishm_tbl->block[i].user_len, + ishm_tbl->block[i].seq, + ishm_tbl->block[i].refcnt); + + if (proc_index < 0) + continue; + + ODP_DBG(" start:%-08lx fd:%-3d\n", + ishm_proctable->entry[proc_index].start, + ishm_proctable->entry[proc_index].fd); + } + + /* display the virtual space allocations... : */ + ODP_DBG("ishm virtual space:\n"); + for (fragmnt = ishm_ftbl->used_fragmnts; + fragmnt; fragmnt = fragmnt->next) { + if (fragmnt->block_index >= 0) { + nb_allocated_frgments++; + ODP_DBG(" %08p - %08p: ALLOCATED by block:%d\n", + (uintptr_t)fragmnt->start, + (uintptr_t)fragmnt->start + fragmnt->len - 1, + fragmnt->block_index); + consecutive_unallocated = 0; + } else { + ODP_DBG(" %08p - %08p: NOT ALLOCATED\n", + (uintptr_t)fragmnt->start, + (uintptr_t)fragmnt->start + fragmnt->len - 1); + if (consecutive_unallocated++) + ODP_ERR("defragmentation error\n"); + } + + /* some other sanity checks: */ + if (fragmnt->prev != previous) + ODP_ERR("chaining error\n"); + + if (fragmnt != ishm_ftbl->used_fragmnts) { + if ((uintptr_t)fragmnt->start != last_address + 1) + ODP_ERR("lost space error\n"); + } + + last_address = (uintptr_t)fragmnt->start + fragmnt->len - 1; + previous = fragmnt; + nb_used_frgments++; + } + + /* + * the number of blocks with the single_VA flag set should match + * the number of used fragments: + */ + if (single_va_blocks != nb_allocated_frgments) + ODP_ERR("single_va_blocks != nb_allocated_fragments!\n"); + + /* compute the number of unused fragments*/ + for (fragmnt = ishm_ftbl->unused_fragmnts; + fragmnt; fragmnt = fragmnt->next) + nb_unused_frgments++; + + ODP_DBG("ishm: %d fragment used. %d fragements unused. (total=%d)\n", + nb_used_frgments, nb_unused_frgments, + nb_used_frgments + nb_unused_frgments); + + if ((nb_used_frgments + nb_unused_frgments) != ISHM_NB_FRAGMNTS) + ODP_ERR("lost fragments!\n"); + + if (nb_blocks < ishm_proctable->nb_entries) + ODP_ERR("process known block cannot exceed main total sum!\n"); + + ODP_DBG("\n"); + + odp_spinlock_unlock(&ishm_tbl->lock); + return nb_blocks; +} diff --git a/platform/linux-generic/include/_ishm_internal.h b/platform/linux-generic/include/_ishm_internal.h index 7d27477..3231b60 100644 --- a/platform/linux-generic/include/_ishm_internal.h +++ b/platform/linux-generic/include/_ishm_internal.h @@ -37,6 +37,7 @@ int _odp_ishm_lookup_by_name(const char *name); int _odp_ishm_lookup_by_address(void *addr); void *_odp_ishm_address(int block_index); int _odp_ishm_info(int block_index, _odp_ishm_info_t *info); +int _odp_ishm_status(const char *title); #ifdef __cplusplus } commit 6ec6cf6347efec4bf4268c6db37f287b9de35e30 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:45:58 2016 +0200 linux-gen: ishm: internal shared memory allocator (ishm) added A new ODP internal memory allocator, called ishm (for internal shmem) is introduced here. This memory allocator enables the following: - works for odpthreads being linux processes, regardless for fork time. - guarantees the uniqueness of the virtual space mapping address over all ODP threads (even processes and regardless of fork time), when the required _ODP_ISHM_SINGLE_VA flag is used. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index cb7c8e9..ac2b2da 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -113,6 +113,8 @@ odpdrvplatinclude_HEADERS = \ noinst_HEADERS = \ ${srcdir}/include/_fdserver_internal.h \ + ${srcdir}/include/_ishm_internal.h \ + ${srcdir}/include/_ishmphy_internal.h \ ${srcdir}/include/odp_align_internal.h \ ${srcdir}/include/odp_atomic_internal.h \ ${srcdir}/include/odp_buffer_inlines.h \ @@ -156,6 +158,8 @@ noinst_HEADERS = \ __LIB__libodp_linux_la_SOURCES = \ _fdserver.c \ + _ishm.c \ + _ishmphy.c \ odp_atomic.c \ odp_barrier.c \ odp_buffer.c \ diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c new file mode 100644 index 0000000..c8156aa --- /dev/null +++ b/platform/linux-generic/_ishm.c @@ -0,0 +1,1335 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/* This file handles the internal shared memory: internal shared memory + * is memory which is sharable by all ODP threads regardless of how the + * ODP thread is implemented (pthread or process) and regardless of fork() + * time. + * Moreover, when reserved with the _ODP_ISHM_SINGLE_VA flag, + * internal shared memory is guaranteed to always be located at the same virtual + * address, i.e. pointers to internal shared memory are fully shareable + * between odp threads (regardless of thread type or fork time) in that case. + * Internal shared memory is mainly meant to be used internaly within ODP + * (hence its name), but may also be allocated by odp applications and drivers, + * in the future (through these interfaces). + * To guarrentee this full pointer shareability (when reserved with the + * _ODP_ISHM_SINGLE_VA flag) internal shared memory is handled as follows: + * At global_init time, a huge virtual address space reservation is performed. + * Note that this is just reserving virtual space, not physical memory. + * Because all ODP threads (pthreads or processes) are descendants of the ODP + * instantiation process, this VA space is inherited by all ODP threads. + * When internal shmem reservation actually occurs, and + * when reserved with the _ODP_ISHM_SINGLE_VA flag, physical memory is + * allocated, and mapped (MAP_FIXED) to some part in the huge preallocated + * address space area: + * because this virtual address space is common to all ODP threads, we + * know this mapping will succeed, and not clash with anything else. + * Hence, an ODP threads which perform a lookup for the same ishm block + * can map it at the same VA address. + * When internal shared memory is released, the physical memory is released + * and the corresponding virtual space returned to its "pool" of preallocated + * virtual space (assuming it was allocated from there). + * Note, though, that, if 2 linux processes share the same ishm block, + * the virtual space is marked as released as soon as one of the processes + * releases the ishm block, but the physical memory space is actually released + * by the kernel once all processes have done a ishm operation (i,e. a sync). + * This is due to the fact that linux does not contain any syscall to unmap + * memory from a different process. + * + * This file contains functions to handle the VA area (handling fragmentation + * and defragmentation resulting from different allocs/release) and also + * define the functions to allocate, release and lookup internal shared + * memory: + * _odp_ishm_reserve(), _odp_ishm_free*() and _odp_ishm_lookup*()... + */ +#include <odp_posix_extensions.h> +#include <odp_config_internal.h> +#include <odp_internal.h> +#include <odp/api/spinlock.h> +#include <odp/api/align.h> +#include <odp/api/system_info.h> +#include <odp/api/debug.h> +#include <odp_shm_internal.h> +#include <odp_debug_internal.h> +#include <odp_align_internal.h> +#include <_fdserver_internal.h> +#include <_ishm_internal.h> +#include <_ishmphy_internal.h> +#include <stdlib.h> +#include <stdio.h> +#include <unistd.h> +#include <string.h> +#include <errno.h> +#include <sys/mman.h> +#include <sys/stat.h> +#include <sys/syscall.h> +#include <fcntl.h> +#include <sys/types.h> +#include <inttypes.h> +#include <sys/wait.h> + +/* + * Maximum number of internal shared memory blocks. + * + * This the the number of separate ISHM areas that can be reserved concurrently + * (Note that freeing such blocks may take time, or possibly never happen + * if some of the block ownwers never procsync() after free). This number + * should take that into account) + */ +#define ISHM_MAX_NB_BLOCKS 128 + +/* + * Maximum internal shared memory block name length in chars + * probably taking the same number as SHM name size make sense at this stage + */ +#define ISHM_NAME_MAXLEN 32 + +/* + * Linux underlying file name: <directory>/odp-<odp_pid>-ishm-<name> + * The <name> part may be replaced by a sequence number if no specific + * name is given at reserve time + * <directory> is either /tmp or the hugepagefs mount point for default size. + * (searched at init time) + */ +#define ISHM_FILENAME_MAXLEN (ISHM_NAME_MAXLEN + 64) +#define ISHM_FILENAME_FORMAT "%s/odp-%d-ishm-%s" +#define ISHM_FILENAME_NORMAL_PAGE_DIR "/tmp" + +/* + * At worse case the virtual space gets so fragmented that there is + * a unallocated fragment between each allocated fragment: + * In that case, the number of fragments to take care of is twice the + * number of ISHM blocks + 1. + */ +#define ISHM_NB_FRAGMNTS (ISHM_MAX_NB_BLOCKS * 2 + 1) + +/* + * A fragment describes a piece of the shared virtual address space, + * and is allocated only when allocation is done with the _ODP_ISHM_SINGLE_VA + * flag: + * A fragment is said to be used when it actually does represent some + * portion of the virtual address space, and is said to be unused when + * it does not (so at start, one single fragment is used -describing the + * whole address space as unallocated-, and all others are unused). + * Fragments get used as address space fragmentation increases. + * A fragment is allocated if the piece of address space it + * describes is actually used by a shared memory block. + * Allocated fragments get their block_index set >=0. + */ +typedef struct ishm_fragment { + struct ishm_fragment *prev; /* not used when the fragment is unused */ + struct ishm_fragment *next; + void *start; /* start of segment (VA) */ + uintptr_t len; /* length of segment. multiple of page size */ + int block_index; /* -1 for unallocated fragments */ +} ishm_fragment_t; + +/* + * A block describes a piece of reserved memory: Any successful ishm_reserve() + * will allocate a block. A ishm_reserve() with the _ODP_ISHM_SINGLE_VA flag set + * will allocate both a block and a fragment. + * Blocks contain only global data common to all processes. + */ +typedef struct ishm_block { + char name[ISHM_NAME_MAXLEN]; /* name for the ishm block (if any) */ + char filename[ISHM_FILENAME_MAXLEN]; /* name of the .../odp-* file */ + int main_odpthread; /* The thread which did the initial reserve*/ + uint32_t user_flags; /* any flags the user want to remember. */ + uint32_t flags; /* block creation flags. */ + uint64_t user_len; /* length, as requested at reserve time. */ + void *start; /* only valid if _ODP_ISHM_SINGLE_VA is set*/ + uint64_t len; /* length. multiple of page size. 0 if free*/ + ishm_fragment_t *fragment; /* used when _ODP_ISHM_SINGLE_VA is used */ + int huge; /* true if this segment is mapped using huge pages */ + uint64_t seq; /* sequence number, incremented on alloc and free */ + uint64_t refcnt;/* number of linux processes mapping this block */ +} ishm_block_t; + +/* + * Table of blocks describing allocated internal shared memory + * This table is visible to every ODP thread (linux process or pthreads). + * (it is allocated shared at odp init time and is therefore inherited by all) + * Table index is used as handle, so it cannot move!. Entry is regarded as + * free when len==0 + */ +typedef struct { + odp_spinlock_t lock; + uint64_t dev_seq; /* used when creating device names */ + ishm_block_t block[ISHM_MAX_NB_BLOCKS]; +} ishm_table_t; +static ishm_table_t *ishm_tbl; + +/* + * Process local table containing the list of (believed) allocated blocks seen + * from the current process. There is one such table per linux process. linux + * threads within a process shares this table. + * The contents within this table may become obsolete when other processes + * reserve/free ishm blocks. This is what the procsync() function + * catches by comparing the block sequence number with the one in this table. + * This table is filled at ishm_reserve and ishm_lookup time. + * Entries are removed at ishm_free or procsync time. + * Note that flags and len are present in this table and seems to be redundant + * with those present in the ishm block table: but this is not fully true: + * When ishm_sync() detects obsolete mappings and tries to remove them, + * the entry in the ishm block table is then obsolete, and the values which are + * found in this table must be used to perform the ummap. + * (and the values in the block tables are needed at lookup time...) + */ +typedef struct { + int thrd_refcnt; /* number of pthreads in this process, really */ + struct { + int block_index; /* entry in the ishm_tbl */ + uint32_t flags; /* flags used at creation time */ + uint64_t seq; + void *start; /* start of block (VA) */ + uint64_t len; /* length of block. multiple of page size */ + int fd; /* file descriptor used for this block */ + } entry[ISHM_MAX_NB_BLOCKS]; + int nb_entries; +} ishm_proctable_t; +static ishm_proctable_t *ishm_proctable; + +/* + * Table of fragments describing the common virtual address space: + * This table is visible to every ODP thread (linux process or pthreads). + * (it is allocated at odp init time and is therefore inherited by all) + */ +typedef struct { + ishm_fragment_t fragment[ISHM_NB_FRAGMNTS]; + ishm_fragment_t *used_fragmnts; /* ordered by increasing start addr */ + ishm_fragment_t *unused_fragmnts; +} ishm_ftable_t; +static ishm_ftable_t *ishm_ftbl; + +#ifndef MAP_ANONYMOUS +#define MAP_ANONYMOUS MAP_ANON +#endif + +/* prototypes: */ +static void procsync(void); + +/* + * Take a piece of the preallocated virtual space to fit "size" bytes. + * (best fit). Size must be rounded up to an integer number of pages size. + * Possibly split the fragment to keep track of remaining space. + * Returns the allocated fragment (best_fragmnt) and the corresponding address. + * External caller must ensure mutex before the call! + */ +static void *alloc_fragment(uintptr_t size, int block_index, intptr_t align, + ishm_fragment_t **best_fragmnt) +{ + ishm_fragment_t *fragmnt; + *best_fragmnt = NULL; + ishm_fragment_t *rem_fragmnt; + uintptr_t border;/* possible start of new fragment (next alignement) */ + intptr_t left; /* room remaining after, if the segment is allocated */ + uintptr_t remainder = ODP_CONFIG_ISHM_VA_PREALLOC_SZ; + + /* + * search for the best bit, i.e. search for the unallocated fragment + * would give less remainder if the new fragment was allocated within + * it: + */ + for (fragmnt = ishm_ftbl->used_fragmnts; + fragmnt; fragmnt = fragmnt->next) { + /* skip allocated segment: */ + if (fragmnt->block_index >= 0) + continue; + /* skip too short segment: */ + border = ((uintptr_t)fragmnt->start + align - 1) & (-align); + left = + ((uintptr_t)fragmnt->start + fragmnt->len) - (border + size); + if (left < 0) + continue; + /* remember best fit: */ + if ((uintptr_t)left < remainder) { + remainder = left; /* best, so far */ + *best_fragmnt = fragmnt; + } + } + + if (!(*best_fragmnt)) { + ODP_ERR("unable to get virtual address for shmem block!\n."); + return NULL; + } + + (*best_fragmnt)->block_index = block_index; + border = ((uintptr_t)(*best_fragmnt)->start + align - 1) & (-align); + + /* + * if there is room between previous fragment and new one, (due to + * alignement requirement) then fragment (split) the space between + * the end of the previous fragment and the beginning of the new one: + */ + if (border - (uintptr_t)(*best_fragmnt)->start > 0) { + /* frangment space, i.e. take a new fragment descriptor... */ + rem_fragmnt = ishm_ftbl->unused_fragmnts; + if (!rem_fragmnt) { + ODP_ERR("unable to get shmem fragment descriptor!\n."); + return NULL; + } + ishm_ftbl->unused_fragmnts = rem_fragmnt->next; + + /* and link it between best_fragmnt->prev and best_fragmnt */ + if ((*best_fragmnt)->prev) + (*best_fragmnt)->prev->next = rem_fragmnt; + else + ishm_ftbl->used_fragmnts = rem_fragmnt; + rem_fragmnt->prev = (*best_fragmnt)->prev; + (*best_fragmnt)->prev = rem_fragmnt; + rem_fragmnt->next = (*best_fragmnt); + + /* update length: rem_fragmnt getting space before border */ + rem_fragmnt->block_index = -1; + rem_fragmnt->start = (*best_fragmnt)->start; + rem_fragmnt->len = border - (uintptr_t)(*best_fragmnt)->start; + (*best_fragmnt)->start = + (void *)((uintptr_t)rem_fragmnt->start + rem_fragmnt->len); + (*best_fragmnt)->len -= rem_fragmnt->len; + } + + /* if this was a perfect fit, i.e. no free space follows, we are done */ + if (remainder == 0) + return (*best_fragmnt)->start; + + /* otherwise, frangment space, i.e. take a new fragment descriptor... */ + rem_fragmnt = ishm_ftbl->unused_fragmnts; + if (!rem_fragmnt) { + ODP_ERR("unable to get shmem fragment descriptor!\n."); + return (*best_fragmnt)->start; + } + ishm_ftbl->unused_fragmnts = rem_fragmnt->next; + + /* ... double link it... */ + rem_fragmnt->next = (*best_fragmnt)->next; + rem_fragmnt->prev = (*best_fragmnt); + if ((*best_fragmnt)->next) + (*best_fragmnt)->next->prev = rem_fragmnt; + (*best_fragmnt)->next = rem_fragmnt; + + /* ... and keep track of the remainder */ + (*best_fragmnt)->len = size; + rem_fragmnt->len = remainder; + rem_fragmnt->start = (void *)((char *)(*best_fragmnt)->start + size); + rem_fragmnt->block_index = -1; + + return (*best_fragmnt)->start; +} + +/* + * Free a portion of virtual space. + * Possibly defragment, if the freed fragment is adjacent to another + * free virtual fragment. + * External caller must ensure mutex before the call! + */ +static void free_fragment(ishm_fragment_t *fragmnt) +{ + ishm_fragment_t *prev_f; + ishm_fragment_t *next_f; + + /* sanity check */ + if (!fragmnt) + return; + + prev_f = fragmnt->prev; + next_f = fragmnt->next; + + /* free the fragment */ + fragmnt->block_index = -1; + + /* check if the previous fragment is also free: if so, defragment */ + if (prev_f && (prev_f->block_index < 0)) { + fragmnt->start = prev_f->start; + fragmnt->len += prev_f->len; + if (prev_f->prev) { + prev_f->prev->next = fragmnt; + } else { + if (ishm_ftbl->used_fragmnts == prev_f) + ishm_ftbl->used_fragmnts = fragmnt; + else + ODP_ERR("corrupted fragment list!.\n"); + } + fragmnt->prev = prev_f->prev; + + /* put removed fragment in free list */ + prev_f->prev = NULL; + prev_f->next = ishm_ftbl->unused_fragmnts; + ishm_ftbl->unused_fragmnts = prev_f; + } + + /* check if the next fragment is also free: if so, defragment */ + if (next_f && (next_f->block_index < 0)) { + fragmnt->len += next_f->len; + if (next_f->next) + next_f->next->prev = fragmnt; + fragmnt->next = next_f->next; + + /* put removed fragment in free list */ + next_f->prev = NULL; + next_f->next = ishm_ftbl->unused_fragmnts; + ishm_ftbl->unused_fragmnts = next_f; + } +} + +/* + * Create file with size len. returns -1 on error + * Creates a file to /tmp/odp-<pid>-<sequence_or_name> (for normal pages) + * or /mnt/huge/odp-<pid>-<sequence_or_name> (for huge pages) + * Return the new file descriptor, or -1 on error. + */ +static int create_file(int block_index, int huge, uint64_t len) +{ + char *name; + int fd; + ishm_block_t *new_block; /* entry in the main block table */ + char seq_string[ISHM_FILENAME_MAXLEN]; /* used to construct filename*/ + char filename[ISHM_FILENAME_MAXLEN];/* filename in /tmp/ or /mnt/huge */ + int oflag = O_RDWR | O_CREAT | O_TRUNC; /* flags for open */ + + new_block = &ishm_tbl->block[block_index]; + name = new_block->name; + + /* create the filename: */ + snprintf(seq_string, ISHM_FILENAME_MAXLEN, "%08" PRIu64, + ishm_tbl->dev_seq++); + + /* huge dir must be known to create files there!: */ + if (huge && !odp_global_data.hugepage_info.default_huge_page_dir) + return -1; + + if (huge) + snprintf(filename, ISHM_FILENAME_MAXLEN, + ISHM_FILENAME_FORMAT, + odp_global_data.hugepage_info.default_huge_page_dir, + odp_global_data.main_pid, + (name && name[0]) ? name : seq_string); + else + snprintf(filename, ISHM_FILENAME_MAXLEN, + ISHM_FILENAME_FORMAT, + ISHM_FILENAME_NORMAL_PAGE_DIR, + odp_global_data.main_pid, + (name && name[0]) ? name : seq_string); + + fd = open(filename, oflag, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); + if (fd < 0) { + ODP_ERR("open failed for %s: %s.\n", + filename, strerror(errno)); + return -1; + } + + if (ftruncate(fd, len) == -1) { + ODP_ERR("ftruncate failed: fd=%d, err=%s.\n", + fd, strerror(errno)); + close(fd); + return -1; + } + + strncpy(new_block->filename, filename, ISHM_FILENAME_MAXLEN - 1); + + return fd; +} + +/* + * performs the mapping, possibly allocating a fragment of the pre-reserved + * VA space if the _ODP_ISHM_SINGLE_VA flag was given. + * Sets fd, and returns the mapping address. + * This funstion will also set the _ODP_ISHM_SINGLE_VA flag if the alignment + * requires it + * Mutex must be assured by the caller. + */ +static void *do_map(int block_index, uint64_t len, uint32_t align, + uint32_t flags, int huge, int *fd) +{ + ishm_block_t *new_block; /* entry in the main block table */ + void *addr = NULL; + void *mapped_addr; + ishm_fragment_t *fragment = NULL; + + new_block = &ishm_tbl->block[block_index]; + + /* + * Creates a file to /tmp/odp-<pid>-<sequence> (for normal pages) + * or /mnt/huge/odp-<pid>-<sequence> (for huge pages) + * unless a fd was already given + */ + if (*fd < 0) { + *fd = create_file(block_index, huge, len); + if (*fd < 0) + return NULL; + } else { + new_block->filename[0] = 0; + } + + /* allocate an address range in the prebooked VA area if needed */ + if (flags & _ODP_ISHM_SINGLE_VA) { + addr = alloc_fragment(len, block_index, align, &fragment); + if (!addr) { + ODP_ERR("alloc_fragment failed.\n"); + if (new_block->filename[0]) { + close(*fd); + *fd = -1; + unlink(new_block->filename); + } + return NULL; + } + ishm_tbl->block[block_index].fragment = fragment; + } + + /* try to mmap: */ + mapped_addr = _odp_ishmphy_map(*fd, addr, len, flags); + if (mapped_addr == NULL) { + if (flags & _ODP_ISHM_SINGLE_VA) + free_fragment(fragment); + if (new_block->filename[0]) { + close(*fd); + *fd = -1; + unlink(new_block->filename); + } + return NULL; + } + + new_block->huge = huge; + + return mapped_addr; +} + +/* + * Performs an extra mapping (for a process trying to see an existing block + * i.e. performing a lookup). + * Mutex must be assured by the caller. + */ +static void *do_remap(int block_index, int fd) +{ + void *mapped_addr; + ishm_fragment_t *fragment; + uint64_t len; + uint32_t flags; + + len = ishm_tbl->block[block_index].len; + flags = ishm_tbl->block[block_index].flags; + + if (flags & _ODP_ISHM_SINGLE_VA) { + fragment = ishm_tbl->block[block_index].fragment; + if (!fragment) { + ODP_ERR("invalid fragment failure.\n"); + return NULL; + } + + /* try to mmap: */ + mapped_addr = _odp_ishmphy_map(fd, fragment->start, len, flags); + if (mapped_addr == NULL) + return NULL; + return mapped_addr; + } + + /* try to mmap: */ + mapped_addr = _odp_ishmphy_map(fd, NULL, len, flags); + if (mapped_addr == NULL) + return NULL; + + return mapped_addr; +} + +/* + * Performs unmapping, possibly freeing a prereserved VA space fragment, + * if the _ODP_ISHM_SINGLE_VA flag was set at alloc time + * Mutex must be assured by the caller. + */ +static int do_unmap(void *start, uint64_t size, uint32_t flags, + int block_index) +{ + int ret; + + if (start) + ret = _odp_ishmphy_unmap(start, size, flags); + else + ret = 0; + + if ((block_index >= 0) && (flags & _ODP_ISHM_SINGLE_VA)) { + /* mark reserved address space as free */ + free_fragment(ishm_tbl->block[block_index].fragment); + } + + return ret; +} + +/* + * Search for a given used and allocated block name. + * (search is performed in the global ishm table) + * Returns the index of the found block (if any) or -1 if none. + * Mutex must be assured by the caller. + */ +static int find_block_by_name(const char *name) +{ + int i; + + if (name == NULL || name[0] == 0) + return -1; + + for (i = 0; i < ISHM_MAX_NB_BLOCKS; i++) { + if ((ishm_tbl->block[i].len) && + (strcmp(name, ishm_tbl->block[i].name) == 0)) + return i; + } + + return -1; +} + +/* + * Search for a block by address (only works when flag _ODP_ISHM_SINGLE_VA + * was set at reserve() time, or if the block is already known by this + * process). + * Search is performed in the process table and in the global ishm table. + * The provided address does not have to be at start: any address + * within the fragment is OK. + * Returns the index to the found block (if any) or -1 if none. + * Mutex must be assured by the caller. + */ +static int find_block_by_address(void *addr) +{ + int block_index; + int i; + ishm_fragment_t *fragmnt; + + /* + * first check if there is already a process known block for this + * address + */ + for (i = 0; i < ishm_proctable->nb_entries; i++) { + block_index = ishm_proctable->entry[i].block_index; + if ((addr > ishm_proctable->entry[i].start) && + ((char *)addr < ((char *)ishm_proctable->entry[i].start + + ishm_tbl->block[block_index].len))) + return block_index; + } + + /* + * then check if there is a existing single VA block known by some other + * process and containing the given address + */ + for (i = 0; i < ISHM_MAX_NB_BLOCKS; i++) { + if ((!ishm_tbl->block[i].len) || + (!(ishm_tbl->block[i].flags & _ODP_ISHM_SINGLE_VA))) + continue; + fragmnt = ishm_tbl->block[i].fragment; + if (!fragmnt) { + ODP_ERR("find_fragment: invalid NULL fragment\n"); + return -1; + } + if ((addr >= fragmnt->start) && + ((char *)addr < ((char *)fragmnt->start + fragmnt->len))) + return i; + } + + /* address does not belong to any accessible block: */ + return -1; +} + +/* + * Search a given ishm block in the process local table. Return its index + * in the process table or -1 if not found (meaning that the ishm table + * block index was not referenced in the process local table, i.e. the + * block is known by some other process, but not by the current process). + * Caller must assure mutex. + */ +static int procfind_block(int block_index) +{ + int i; + + for (i = 0; i < ishm_proctable->nb_entries; i++) { + if (ishm_proctable->entry[i].block_index == block_index) + return i; + } + return -1; +} + +/* + * Release the physical memory mapping for blocks which have been freed + * by other processes. Caller must ensure mutex. + * Mutex must be assured by the caller. + */ +static void procsync(void) +{ + int i = 0; + int last; + ishm_block_t *block; + + last = ishm_proctable->nb_entries; + while (i < last) { + /* if the procecess sequence number doesn't match the main + * table seq number, this entry is obsolete + */ + block = &ishm_tbl->block[ishm_proctable->entry[i].block_index]; + if (ishm_proctable->entry[i].seq != block->seq) { + /* obsolete entry: free memory and remove proc entry */ + close(ishm_proctable->entry[i].fd); + _odp_ishmphy_unmap(ishm_proctable->entry[i].start, + ishm_proctable->entry[i].len, + ishm_proctable->entry[i].flags); + ishm_proctable->entry[i] = + ishm_proctable->entry[--last]; + } else { + i++; + } + } + ishm_proctable->nb_entries = last; +} + +/* + * Allocate and map internal shared memory, or other objects: + * If a name is given, check that this name is not already in use. + * If ok, allocate a new shared memory block and map the + * provided fd in it (if fd >=0 was given). + * If no fd is provided, a shared memory file desc named + * /tmp/odp-<pid>-ishm-<name_or_sequence> is created and mapped. + * (the name is different for huge page file as they must be on hugepagefs) + * The function returns the index of the newly created block in the + * main block table (>=0) or -1 on error. + */ +int _odp_ishm_reserve(const char *name, uint64_t size, int fd, + uint32_t align, uint32_t flags, uint32_t user_flags) +{ + int new_index; /* index in the main block table*/ + ishm_block_t *new_block; /* entry in the main block table*/ + uint64_t page_sz; /* normal page size. usually 4K*/ + uint64_t alloc_size; /* includes extra for alignement*/ + uint64_t page_hp_size; /* huge page size */ + uint64_t alloc_hp_size; /* includes extra for alignement*/ + uint32_t hp_align; + uint64_t len; /* mapped length */ + void *addr = NULL; /* mapping address */ + int new_proc_entry; + + page_sz = odp_sys_page_size(); + + odp_spinlock_lock(&ishm_tbl->lock); + + /* update this process view... */ + procsync(); + + /* roundup to page size */ + alloc_size = (size + (page_sz - 1)) & (-page_sz); + + page_hp_size = odp_sys_huge_page_size(); + /* roundup to page size */ + alloc_hp_size = (size + (page_hp_size - 1)) & (-page_hp_size); + + /* check if name already exists */ + if (name && (find_block_by_name(name) >= 0)) { + /* Found a block with the same name */ + odp_spinlock_unlock(&ishm_tbl->lock); + ODP_ERR("name \"%s\" already used.\n", name); + return -1; + } + + /* grab a new entry: */ + for (new_index = 0; new_index < ISHM_MAX_NB_BLOCKS; new_index++) { + if (ishm_tbl->block[new_index].len == 0) { + /* Found free block */ + break; + } + } + + /* check if we have reached the maximum number of allocation: */ + if (new_index >= ISHM_MAX_NB_BLOCKS) { + odp_spinlock_unlock(&ishm_tbl->lock); + ODP_ERR("ISHM_MAX_NB_BLOCKS limit reached!\n"); + return -1; + } + + new_block = &ishm_tbl->block[new_index]; + + /* save block name (if any given): */ + if (name) + strncpy(new_block->name, name, ISHM_NAME_MAXLEN - 1); + else + new_block->name[0] = 0; + + /* Try first huge pages when possible and needed: */ + if (page_hp_size && (alloc_size > page_sz)) { + /* at least, alignment in VA should match page size, but user + * can request more: If the user requirement exceeds the page + * size then we have to make sure the block will be mapped at + * the same address every where, otherwise alignment may be + * be wrong for some process */ + hp_align = align; + if (hp_align < odp_sys_huge_page_size()) + hp_align = odp_sys_huge_page_size(); + else + flags |= _ODP_ISHM_SINGLE_VA; + len = alloc_hp_size; + addr = do_map(new_index, len, hp_align, flags, 1, &fd); + + if (addr == NULL) + ODP_DBG("No huge pages, fall back to normal pages, " + "check: /proc/sys/vm/nr_hugepages.\n"); + else + new_block->huge = 1; + } + + /* try normal pages if huge pages failed */ + if (addr == NULL) { + /* at least, alignment in VA should match page size, but user + * can request more: If the user requirement exceeds the page + * size then we have to make sure the block will be mapped at + * the same address every where, otherwise alignment may be + * be wrong for some process */ + if (align < odp_sys_page_size()) + align = odp_sys_page_size(); + else + flags |= _ODP_ISHM_SINGLE_VA; + + len = alloc_size; + addr = do_map(new_index, len, align, flags, 0, &fd); + new_block->huge = 0; + } + + /* if neither huge pages or normal pages works, we cannot proceed: */ + if ((addr == NULL) || (len == 0)) { + if ((new_block->filename[0]) && (fd >= 0)) + close(fd); + odp_spinlock_unlock(&ishm_tbl->lock); + ODP_ERR("_ishm_reserve failed.\n"); + return -1; + } + + /* remember block data and increment block seq number to mark change */ + new_block->len = len; + new_block->user_len = size; + new_block->flags = flags; + new_block->user_flags = user_flags; + new_block->seq++; + new_block->refcnt = 1; + new_block->main_odpthread = odp_thread_id(); + new_block->start = addr; /* only for SINGLE_VA*/ + + /* the allocation succeeded: update the process local view */ + new_proc_entry = ishm_proctable->nb_entries++; + ishm_proctable->entry[new_proc_entry].block_index = new_index; + ishm_proctable->entry[new_proc_entry].flags = flags; + ishm_proctable->entry[new_proc_entry].seq = new_block->seq; + ishm_proctable->entry[new_proc_entry].start = addr; + ishm_proctable->entry[new_proc_entry].len = len; + ishm_proctable->entry[new_proc_entry].fd = fd; + + /* register the file descriptor to the file descriptor server. */ + _odp_fdserver_register_fd(FD_SRV_CTX_ISHM, new_index, fd); + + odp_spinlock_unlock(&ishm_tbl->lock); + return new_index; +} + +/* + * Free and unmap internal shared memory: + * The file descriptor is closed and the .../odp-* file deleted, + * unless fd was externally provided at reserve() time. + * return 0 if OK, and -1 on error. + * Mutex must be assured by the caller. + */ +static int block_free(int block_index) +{ + int proc_index; + ishm_block_t *block; /* entry in the main block table*/ + int last; + + if ((block_index < 0) || + (block_index >= ISHM_MAX_NB_BLOCKS) || + (ishm_tbl->block[block_index].len == 0)) { + ODP_ERR("Request to free an invalid block\n"); + return -1; + } + + block = &ishm_tbl->block[block_index]; + + proc_index = procfind_block(block_index); + if (proc_index >= 0) { + /* close the fd, unless if it was externaly provided */ + if ((block->filename[0] != 0) || + (odp_thread_id() != block->main_odpthread)) + close(ishm_proctable->entry[proc_index].fd); + + /* remove the mapping and possible fragment */ + do_unmap(ishm_proctable->entry[proc_index].start, + block->len, + ishm_proctable->entry[proc_index].flags, + block_index); + + /* remove entry from process local table: */ + last = ishm_proctable->nb_entries - 1; + ishm_proctable->entry[proc_index] = + ishm_proctable->entry[last]; + ishm_proctable->nb_entries = last; + } else { + /* just possibly free the fragment as no mapping exist here: */ + do_unmap(NULL, 0, block->flags, block_index); + } + + /* remove the .../odp-* file, unless fd was external: */ + if (block->filename[0] != 0) + unlink(block->filename); + + /* deregister the file descriptor from the file descriptor server. */ + _odp_fdserver_deregister_fd(FD_SRV_CTX_ISHM, block_index); + + /* mark the block as free in the main block table: */ + block->len = 0; + + /* mark the change so other processes see this entry as obsolete: */ + block->seq++; + + return 0; +} + +/* + * Free and unmap internal shared memory, intentified by its block number: + * return -1 on error. 0 if OK. + */ +int _odp_ishm_free_by_index(int block_index) +{ + int ret; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + ret = block_free(block_index); + odp_spinlock_unlock(&ishm_tbl->lock); + return ret; +} + +/* + * free and unmap internal shared memory, intentified by its block name: + * return -1 on error. 0 if OK. + */ +int _odp_ishm_free_by_name(const char *name) +{ + int block_index; + int ret; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* search the block in main ishm table */ + block_index = find_block_by_name(name); + if (block_index < 0) { + ODP_ERR("Request to free an non existing block..." + " (double free?)\n"); + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + ret = block_free(block_index); + odp_spinlock_unlock(&ishm_tbl->lock); + return ret; +} + +/* + * Free and unmap internal shared memory identified by address: + * return -1 on error. 0 if OK. + */ +int _odp_ishm_free_by_address(void *addr) +{ + int block_index; + int ret; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* search the block in main ishm table */ + block_index = find_block_by_address(addr); + if (block_index < 0) { + ODP_ERR("Request to free an non existing block..." + " (double free?)\n"); + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + ret = block_free(block_index); + + odp_spinlock_unlock(&ishm_tbl->lock); + return ret; +} + +/* + * Lookup for an ishm shared memory, identified by its block index + * in the main ishm block table. + * Map this ishm area in the process VA (if not already present). + * Returns the block user address or NULL on error. + * Mutex must be assured by the caller. + */ +static void *block_lookup(int block_index) +{ + int proc_index; + int fd = -1; + ishm_block_t *block; + void *mapped_addr; + int new_entry; + + if ((block_index < 0) || + (block_index >= ISHM_MAX_NB_BLOCKS) || + (ishm_tbl->block[block_index].len == 0)) { + ODP_ERR("Request to lookup an invalid block\n"); + return NULL; + } + + /* search it in process table: if there, this process knows it already*/ + proc_index = procfind_block(block_index); + if (proc_index >= 0) + return ishm_proctable->entry[proc_index].start; + + /* this ishm is not known by this process, yet: we create the mapping.*/ + fd = _odp_fdserver_lookup_fd(FD_SRV_CTX_ISHM, block_index); + if (fd < 0) { + ODP_ERR("Could not find ishm file descriptor (BUG!)\n"); + return NULL; + } + + /* perform the mapping */ + block = &ishm_tbl->block[block_index]; + + mapped_addr = do_remap(block_index, fd); + if (mapped_addr == NULL) { + ODP_ERR(" lookup: Could not map existing shared memory!\n"); + return NULL; + } + + /* the mapping succeeded: update the process local view */ + new_entry = ishm_proctable->nb_entries++; + ishm_proctable->entry[new_entry].block_index = block_index; + ishm_proctable->entry[new_entry].flags = block->flags; + ishm_proctable->entry[new_entry].seq = block->seq; + ishm_proctable->entry[new_entry].start = mapped_addr; + ishm_proctable->entry[new_entry].len = block->len; + ishm_proctable->entry[new_entry].fd = fd; + block->refcnt++; + + return mapped_addr; +} + +/* + * Lookup for an ishm shared memory, identified by its block_index. + * Maps this ishmem area in the process VA (if not already present). + * Returns the block user address, or NULL if the index + * does not match any known ishm blocks. + */ +void *_odp_ishm_lookup_by_index(int block_index) +{ + void *ret; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + ret = block_lookup(block_index); + odp_spinlock_unlock(&ishm_tbl->lock); + return ret; +} + +/* + * Lookup for an ishm shared memory, identified by its block name. + * Map this ishm area in the process VA (if not already present). + * Return the block index, or -1 if the index + * does not match any known ishm blocks. + */ +int _odp_ishm_lookup_by_name(const char *name) +{ + int block_index; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* search the block in main ishm table: return -1 if not found: */ + block_index = find_block_by_name(name); + if ((block_index < 0) || (!block_lookup(block_index))) { + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + odp_spinlock_unlock(&ishm_tbl->lock); + return block_index; +} + +/* + * Lookup for an ishm shared memory block, identified by its VA address. + * This works only if the block has already been looked-up (mapped) by the + * current process or it it was created with the _ODP_ISHM_SINGLE_VA flag. + * Map this ishm area in the process VA (if not already present). + * Return the block index, or -1 if the address + * does not match any known ishm blocks. + */ +int _odp_ishm_lookup_by_address(void *addr) +{ + int block_index; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* search the block in main ishm table: return -1 if not found: */ + block_index = find_block_by_address(addr); + if ((block_index < 0) || (!block_lookup(block_index))) { + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + odp_spinlock_unlock(&ishm_tbl->lock); + return block_index; +} + +/* + * Returns the VA address of a given block (which has to be known in the current + * process). Returns NULL if the block is unknown. + */ +void *_odp_ishm_address(int block_index) +{ + int proc_index; + void *addr; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + if ((block_index < 0) || + (block_index >= ISHM_MAX_NB_BLOCKS) || + (ishm_tbl->block[block_index].len == 0)) { + ODP_ERR("Request for address on an invalid block\n"); + odp_spinlock_unlock(&ishm_tbl->lock); + return NULL; + } + + proc_index = procfind_block(block_index); + if (proc_index < 0) { + odp_spinlock_unlock(&ishm_tbl->lock); + return NULL; + } + + addr = ishm_proctable->entry[proc_index].start; + odp_spinlock_unlock(&ishm_tbl->lock); + return addr; +} + +int _odp_ishm_info(int block_index, _odp_ishm_info_t *info) +{ + int proc_index; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + if ((block_index < 0) || + (block_index >= ISHM_MAX_NB_BLOCKS) || + (ishm_tbl->block[block_index].len == 0)) { + odp_spinlock_unlock(&ishm_tbl->lock); + ODP_ERR("Request for info on an invalid block\n"); + return -1; + } + + /* search it in process table: if not there, need to map*/ + proc_index = procfind_block(block_index); + if (proc_index < 0) { + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + info->name = ishm_tbl->block[block_index].name; + info->addr = ishm_proctable->entry[proc_index].start; + info->size = ishm_tbl->block[block_index].user_len; + info->page_size = ishm_tbl->block[block_index].huge ? + odp_sys_huge_page_size() : odp_sys_page_size(); + info->flags = ishm_tbl->block[block_index].flags; + info->user_flags = ishm_tbl->block[block_index].user_flags; + + odp_spinlock_unlock(&ishm_tbl->lock); + return 0; +} + +int _odp_ishm_init_global(void) +{ + void *addr; + void *spce_addr; + int i; + + if (!odp_global_data.hugepage_info.default_huge_page_dir) + ODP_DBG("NOTE: No support for huge pages\n"); + else + ODP_DBG("Huge pages mount point is: %s\n", + odp_global_data.hugepage_info.default_huge_page_dir); + + /* allocate space for the internal shared mem block table: */ + addr = mmap(NULL, sizeof(ishm_table_t), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); + if (addr == MAP_FAILED) { + ODP_ERR("unable to mmap the main block table\n."); + goto init_glob_err1; + } + ishm_tbl = addr; + memset(ishm_tbl, 0, sizeof(ishm_table_t)); + ishm_tbl->dev_seq = 0; + odp_spinlock_init(&ishm_tbl->lock); + + /* allocate space for the internal shared mem fragment table: */ + addr = mmap(NULL, sizeof(ishm_ftable_t), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); + if (addr == MAP_FAILED) { + ODP_ERR("unable to mmap the main fragment table\n."); + goto init_glob_err2; + } + ishm_ftbl = addr; + memset(ishm_ftbl, 0, sizeof(ishm_ftable_t)); + + /* + *reserve the address space for _ODP_ISHM_SINGLE_VA reserved blocks, + * only address space! + */ + spce_addr = _odp_ishmphy_book_va(ODP_CONFIG_ISHM_VA_PREALLOC_SZ, + odp_sys_huge_page_size()); + if (!spce_addr) { + ODP_ERR("unable to reserve virtual space\n."); + goto init_glob_err3; + } + + /* use the first fragment descriptor to describe to whole VA space: */ + ishm_ftbl->fragment[0].block_index = -1; + ishm_ftbl->fragment[0].start = spce_addr; + ishm_ftbl->fragment[0].len = ODP_CONFIG_ISHM_VA_PREALLOC_SZ; + ishm_ftbl->fragment[0].prev = NULL; + ishm_ftbl->fragment[0].next = NULL; + ishm_ftbl->used_fragmnts = &ishm_ftbl->fragment[0]; + + /* and put all other fragment descriptors in the unused list: */ + for (i = 1; i < ISHM_NB_FRAGMNTS - 1; i++) { + ishm_ftbl->fragment[i].prev = NULL; + ishm_ftbl->fragment[i].next = &ishm_ftbl->fragment[i + 1]; + } + ishm_ftbl->fragment[ISHM_NB_FRAGMNTS - 1].prev = NULL; + ishm_ftbl->fragment[ISHM_NB_FRAGMNTS - 1].next = NULL; + ishm_ftbl->unused_fragmnts = &ishm_ftbl->fragment[1]; + + return 0; + +init_glob_err3: + if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0) + ODP_ERR("unable to munmap main fragment table\n."); +init_glob_err2: + if (munmap(ishm_tbl, sizeof(ishm_table_t)) < 0) + ODP_ERR("unable to munmap main block table\n."); +init_glob_err1: + return -1; +} + +int _odp_ishm_init_local(void) +{ + int i; + int block_index; + + /* + * the ishm_process table is local to each linux process + * Check that no other linux threads (of same or ancestor processes) + * have already created the table, and create it if needed. + * We protect this with the general ishm lock to avoid + * init race condition of different running threads. + */ + odp_spinlock_lock(&ishm_tbl->lock); + if (!ishm_proctable) { + ishm_proctable = malloc(sizeof(ishm_proctable_t)); + if (!ishm_proctable) { + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + memset(ishm_proctable, 0, sizeof(ishm_proctable_t)); + } + if (syscall(SYS_gettid) != getpid()) + ishm_proctable->thrd_refcnt++; /* new linux thread */ + else + ishm_proctable->thrd_refcnt = 1;/* new linux process */ + + /* + * if this ODP thread is actually a new linux process, (as opposed + * to a pthread), i.e, we just forked, then all shmem blocks + * of the parent process are mapped into this child by inheritance. + * (The process local table is inherited as well). We hence have to + * increase the process refcount for each of the inherited mappings: + */ + if (syscall(SYS_gettid) == getpid()) { + for (i = 0; i < ishm_proctable->nb_entries; i++) { + block_index = ishm_proctable->entry[i].block_index; + ishm_tbl->block[block_index].refcnt++; + } + } + + odp_spinlock_unlock(&ishm_tbl->lock); + return 0; +} + +int _odp_ishm_term_global(void) +{ + int ret = 0; + + /* free the fragment table */ + if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0) { + ret = -1; + ODP_ERR("unable to munmap fragment table\n."); + } + /* free the block table */ + if (munmap(ishm_tbl, sizeof(ishm_table_t)) < 0) { + ret = -1; + ODP_ERR("unable to munmap main table\n."); + } + + /* free the reserved VA space */ + if (_odp_ishmphy_unbook_va()) + ret = -1; + + return ret; +} + +int _odp_ishm_term_local(void) +{ + int i; + int proc_table_refcnt = 0; + int block_index; + ishm_block_t *block; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* + * The ishm_process table is local to each linux process + * Check that no other linux threads (of this linux process) + * still needs the table, and free it if so. + * We protect this with the general ishm lock to avoid + * term race condition of different running threads. + */ + proc_table_refcnt = --ishm_proctable->thrd_refcnt; + if (!proc_table_refcnt) { + /* + * this is the last thread of this process... + * All mappings for this process are about to be lost... + * Go through the table of visible blocks for this process, + * decreasing the refcnt of each visible blocks, and issuing + * warning for those no longer referenced by any process. + * Note that non-referenced blocks are nor freeed: this is + * deliberate as this would imply that the sementic of the + * freeing function would differ depending on whether we run + * with odp_thread as processes or pthreads. With this approach, + * the user should always free the blocks manually, which is + * more consistent + */ + for (i = 0; i < ishm_proctable->nb_entries; i++) { + block_index = ishm_proctable->entry[i].block_index; + block = &ishm_tbl->block[block_index]; + if ((--block->refcnt) <= 0) { + block->refcnt = 0; + ODP_DBG("Warning: block %d: name:%s " + "no longer referenced\n", + i, + ishm_tbl->block[i].name[0] ? + ishm_tbl->block[i].name : "<no name>"); + } + } + + free(ishm_proctable); + ishm_proctable = NULL; + } + + odp_spinlock_unlock(&ishm_tbl->lock); + return 0; +} diff --git a/platform/linux-generic/_ishmphy.c b/platform/linux-generic/_ishmphy.c new file mode 100644 index 0000000..2b2d100 --- /dev/null +++ b/platform/linux-generic/_ishmphy.c @@ -0,0 +1,185 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/* + * This file handles the lower end of the ishm memory allocator: + * It performs the physical mappings. + */ +#include <odp_posix_extensions.h> +#include <odp_config_internal.h> +#include <odp_internal.h> +#include <odp/api/align.h> +#include <odp/api/system_info.h> +#include <odp/api/debug.h> +#include <odp_debug_internal.h> +#include <odp_align_internal.h> +#include <_ishm_internal.h> +#include <_ishmphy_internal.h> + +#include <stdlib.h> +#include <stdio.h> +#include <unistd.h> +#include <string.h> +#include <errno.h> +#include <sys/mman.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <sys/types.h> +#include <sys/wait.h> +#include <_ishmphy_internal.h> + +static void *common_va_address; +static uint64_t common_va_len; + +#ifndef MAP_ANONYMOUS +#define MAP_ANONYMOUS MAP_ANON +#endif + +/* Book some virtual address space + * This function is called at odp_init_global() time to pre-book some + * virtual address space inherited by all odpthreads (i.e. descendant + * processes and threads) and later used to guarantee the unicity the + * the mapping VA address when memory is reserver with the _ODP_ISHM_SINGLE_VA + * flag. + * returns the address of the mapping or NULL on error. + */ +void *_odp_ishmphy_book_va(uintptr_t len, intptr_t align) +{ + void *addr; + + addr = mmap(NULL, len + align, PROT_NONE, + MAP_SHARED | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + if (addr == MAP_FAILED) { + ODP_ERR("_ishmphy_book_va failure\n"); + return NULL; + } + + if (mprotect(addr, len, PROT_NONE)) + ODP_ERR("failure for protect\n"); + + ODP_DBG("VA Reserved: %p, len=%p\n", addr, len + align); + + common_va_address = addr; + common_va_len = len; + + /* return the nearest aligned address: */ + return (void *)(((uintptr_t)addr + align - 1) & (-align)); +} + +/* Un-book some virtual address space + * This function is called at odp_term_global() time to unbook + * the virtual address space booked by _ishmphy_book_va() + */ +int _odp_ishmphy_unbook_va(void) +{ + int ret; + + ret = munmap(common_va_address, common_va_len); + if (ret) + ODP_ERR("_unishmphy_book_va failure\n"); + return ret; +} + +/* + * do a mapping: + * Performs a mapping of the provided file descriptor to the process VA + * space. If the _ODP_ISHM_SINGLE_VA flag is set, 'start' is assumed to be + * the VA address where the mapping is to be done. + * If the flag is not set, a new VA address is taken. + * returns the address of the mapping or NULL on error. + */ +void *_odp_ishmphy_map(int fd, void *start, uint64_t size, + int flags) +{ + void *mapped_addr; + int mmap_flags = 0; + + if (flags & _ODP_ISHM_SINGLE_VA) { + if (!start) { + ODP_ERR("failure: missing address\n"); + return NULL; + } + /* maps over fragment of reserved VA: */ + mapped_addr = mmap(start, size, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED | mmap_flags, fd, 0); + /* if mapping fails, re-block the space we tried to take + * as it seems a mapping failure still affect what was there??*/ + if (mapped_addr == MAP_FAILED) { + mmap_flags = MAP_SHARED | MAP_FIXED | + MAP_ANONYMOUS | MAP_NORESERVE; + mmap(start, size, PROT_NONE, mmap_flags, -1, 0); + mprotect(start, size, PROT_NONE); + } + } else { + /* just do a new mapping in the VA space: */ + mapped_addr = mmap(NULL, size, PROT_READ | PROT_WRITE, + MAP_SHARED | mmap_flags, fd, 0); + if ((mapped_addr >= common_va_address) && + ((char *)mapped_addr < + (char *)common_va_address + common_va_len)) { + ODP_ERR("VA SPACE OVERLAP!\n"); + } + } + + if (mapped_addr == MAP_FAILED) { + ODP_ERR("mmap failed:%s\n", strerror(errno)); + return NULL; + } + + /* if locking is requested, lock it...*/ + if (flags & _ODP_ISHM_LOCK) { + if (mlock(mapped_addr, size)) { + if (munmap(mapped_addr, size)) + ODP_ERR("munmap failed:%s\n", strerror(errno)); + ODP_ERR("mlock failed:%s\n", strerror(errno)); + return NULL; + } + } + return mapped_addr; +} + +/* free a mapping: + * If the _ODP_ISHM_SINGLE_VA flag was given at creation time the virtual + * address range must be returned to the preoallocated "pool". this is + * done by mapping non accessibly memory there (hence blocking the VA but + * releasing the physical memory). + * If the _ODP_ISHM_SINGLE_VA flag was not given, both physical memory and + * virtual address space are realeased by calling the normal munmap. + * return 0 on success or -1 on error. + */ +int _odp_ishmphy_unmap(void *start, uint64_t len, int flags) +{ + void *addr; + int ret; + int mmap_flgs; + + mmap_flgs = MAP_SHARED | MAP_FIXED | MAP_ANONYMOUS | MAP_NORESERVE; + + /* if locking was requested, unlock...*/ + if (flags & _ODP_ISHM_LOCK) + munlock(start, len); + + if (flags & _ODP_ISHM_SINGLE_VA) { + /* map unnaccessible memory overwrites previous mapping + * and free the physical memory, but guarantees to block + * the VA range from other mappings + */ + addr = mmap(start, len, PROT_NONE, mmap_flgs, -1, 0); + if (addr == MAP_FAILED) { + ODP_ERR("_ishmphy_free failure for ISHM_SINGLE_VA\n"); + return -1; + } + if (mprotect(start, len, PROT_NONE)) + ODP_ERR("_ishmphy_free failure for protect\n"); + return 0; + } + + /* just release the mapping */ + ret = munmap(start, len); + if (ret) + ODP_ERR("_ishmphy_free failure: %s\n", strerror(errno)); + return ret; +} diff --git a/platform/linux-generic/arch/arm/odp/api/cpu_arch.h b/platform/linux-generic/arch/arm/odp/api/cpu_arch.h deleted file mode 120000 index e86e132..0000000 --- a/platform/linux-generic/arch/arm/odp/api/cpu_arch.h +++ /dev/null @@ -1 +0,0 @@ -../../../default/odp/api/cpu_arch.h \ No newline at end of file diff --git a/platform/linux-generic/arch/arm/odp/api/cpu_arch.h b/platform/linux-generic/arch/arm/odp/api/cpu_arch.h new file mode 100644 index 0000000..22b1da2 --- /dev/null +++ b/platform/linux-generic/arch/arm/odp/api/cpu_arch.h @@ -0,0 +1,24 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef ODP_PLAT_CPU_ARCH_H_ +#define ODP_PLAT_CPU_ARCH_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#define _ODP_CACHE_LINE_SIZE 64 + +static inline void odp_cpu_pause(void) +{ +} + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/arch/arm/odp_cpu_arch.c b/platform/linux-generic/arch/arm/odp_cpu_arch.c deleted file mode 120000 index deebc47..0000000 --- a/platform/linux-generic/arch/arm/odp_cpu_arch.c +++ /dev/null @@ -1 +0,0 @@ -../default/odp_cpu_arch.c \ No newline at end of file diff --git a/platform/linux-generic/arch/arm/odp_cpu_arch.c b/platform/linux-generic/arch/arm/odp_cpu_arch.c new file mode 100644 index 0000000..2ac223e --- /dev/null +++ b/platform/linux-generic/arch/arm/odp_cpu_arch.c @@ -0,0 +1,48 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp_posix_extensions.h> + +#include <stdlib.h> +#include <time.h> + +#include <odp/api/cpu.h> +#include <odp/api/hints.h> +#include <odp/api/system_info.h> +#include <odp_debug_internal.h> + +#define GIGA 1000000000 + +uint64_t odp_cpu_cycles(void) +{ + struct timespec time; + uint64_t sec, ns, hz, cycles; + int ret; + + ret = clock_gettime(CLOCK_MONOTONIC_RAW, &time); + + if (ret != 0) + ODP_ABORT("clock_gettime failed\n"); + + hz = odp_cpu_hz_max(); + sec = (uint64_t)time.tv_sec; + ns = (uint64_t)time.tv_nsec; + + cycles = sec * hz; + cycles += (ns * hz) / GIGA; + + return cycles; +} + +uint64_t odp_cpu_cycles_max(void) +{ + return UINT64_MAX; +} + +uint64_t odp_cpu_cycles_resolution(void) +{ + return 1; +} diff --git a/platform/linux-generic/arch/arm/odp_sysinfo_parse.c b/platform/linux-generic/arch/arm/odp_sysinfo_parse.c deleted file mode 120000 index 39962b8..0000000 --- a/platform/linux-generic/arch/arm/odp_sysinfo_parse.c +++ /dev/null @@ -1 +0,0 @@ -../default/odp_sysinfo_parse.c \ No newline at end of file diff --git a/platform/linux-generic/arch/arm/odp_sysinfo_parse.c b/platform/linux-generic/arch/arm/odp_sysinfo_parse.c new file mode 100644 index 0000000..53e2aae --- /dev/null +++ b/platform/linux-generic/arch/arm/odp_sysinfo_parse.c @@ -0,0 +1,27 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp_internal.h> +#include <odp_debug_internal.h> +#include <string.h> + +int cpuinfo_parser(FILE *file ODP_UNUSED, system_info_t *sysinfo) +{ + int i; + + ODP_DBG("Warning: use dummy values for freq and model string\n"); + for (i = 0; i < MAX_CPU_NUMBER; i++) { + sysinfo->cpu_hz_max[i] = 1400000000; + strcpy(sysinfo->model_str[i], "UNKNOWN"); + } + + return 0; +} + +uint64_t odp_cpu_hz_current(int id ODP_UNUSED) +{ + return 0; +} diff --git a/platform/linux-generic/arch/powerpc/odp_cpu_arch.c b/platform/linux-generic/arch/powerpc/odp_cpu_arch.c deleted file mode 120000 index deebc47..0000000 --- a/platform/linux-generic/arch/powerpc/odp_cpu_arch.c +++ /dev/null @@ -1 +0,0 @@ -../default/odp_cpu_arch.c \ No newline at end of file diff --git a/platform/linux-generic/arch/powerpc/odp_cpu_arch.c b/platform/linux-generic/arch/powerpc/odp_cpu_arch.c new file mode 100644 index 0000000..2ac223e --- /dev/null +++ b/platform/linux-generic/arch/powerpc/odp_cpu_arch.c @@ -0,0 +1,48 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp_posix_extensions.h> + +#include <stdlib.h> +#include <time.h> + +#include <odp/api/cpu.h> +#include <odp/api/hints.h> +#include <odp/api/system_info.h> +#include <odp_debug_internal.h> + +#define GIGA 1000000000 + +uint64_t odp_cpu_cycles(void) +{ + struct timespec time; + uint64_t sec, ns, hz, cycles; + int ret; + + ret = clock_gettime(CLOCK_MONOTONIC_RAW, &time); + + if (ret != 0) + ODP_ABORT("clock_gettime failed\n"); + + hz = odp_cpu_hz_max(); + sec = (uint64_t)time.tv_sec; + ns = (uint64_t)time.tv_nsec; + + cycles = sec * hz; + cycles += (ns * hz) / GIGA; + + return cycles; +} + +uint64_t odp_cpu_cycles_max(void) +{ + return UINT64_MAX; +} + +uint64_t odp_cpu_cycles_resolution(void) +{ + return 1; +} diff --git a/platform/linux-generic/include/_fdserver_internal.h b/platform/linux-generic/include/_fdserver_internal.h index 480ac02..22b2802 100644 --- a/platform/linux-generic/include/_fdserver_internal.h +++ b/platform/linux-generic/include/_fdserver_internal.h @@ -23,6 +23,7 @@ extern "C" { */ typedef enum fd_server_context { FD_SRV_CTX_NA, /* Not Applicable */ + FD_SRV_CTX_ISHM, FD_SRV_CTX_END, /* upper enum limit */ } fd_server_context_e; diff --git a/platform/linux-generic/include/_ishm_internal.h b/platform/linux-generic/include/_ishm_internal.h new file mode 100644 index 0000000..7d27477 --- /dev/null +++ b/platform/linux-generic/include/_ishm_internal.h @@ -0,0 +1,45 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef ODP_ISHM_INTERNAL_H_ +#define ODP_ISHM_INTERNAL_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +/* flags available at ishm_reserve: */ +#define _ODP_ISHM_SINGLE_VA 1 +#define _ODP_ISHM_LOCK 2 + +/** + * Shared memory block info + */ +typedef struct _odp_ishm_info_t { + const char *name; /**< Block name */ + void *addr; /**< Block address */ + uint64_t size; /**< Block size in bytes */ + uint64_t page_size; /**< Memory page size */ + uint32_t flags; /**< _ODP_ISHM_* flags */ + uint32_t user_flags;/**< user specific flags */ +} _odp_ishm_info_t; + +int _odp_ishm_reserve(const char *name, uint64_t size, int fd, uint32_t align, + uint32_t flags, uint32_t user_flags); +int _odp_ishm_free_by_index(int block_index); +int _odp_ishm_free_by_name(const char *name); +int _odp_ishm_free_by_address(void *addr); +void *_odp_ishm_lookup_by_index(int block_index); +int _odp_ishm_lookup_by_name(const char *name); +int _odp_ishm_lookup_by_address(void *addr); +void *_odp_ishm_address(int block_index); +int _odp_ishm_info(int block_index, _odp_ishm_info_t *info); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/_ishmphy_internal.h b/platform/linux-generic/include/_ishmphy_internal.h new file mode 100644 index 0000000..4fe560f --- /dev/null +++ b/platform/linux-generic/include/_ishmphy_internal.h @@ -0,0 +1,25 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ISHMPHY_INTERNAL_H +#define _ISHMPHY_INTERNAL_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +void *_odp_ishmphy_book_va(uintptr_t len, intptr_t align); +int _odp_ishmphy_unbook_va(void); +void *_odp_ishmphy_map(int fd, void *start, uint64_t size, int flags); +int _odp_ishmphy_unmap(void *start, uint64_t len, int flags); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/ishmphy_internal.h b/platform/linux-generic/include/ishmphy_internal.h new file mode 100644 index 0000000..0bc4207 --- /dev/null +++ b/platform/linux-generic/include/ishmphy_internal.h @@ -0,0 +1,24 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ISHMPHY_INTERNAL_H_ +#define _ISHMPHY_INTERNAL_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +void *_ishmphy_book_va(uint64_t len); +int _ishmphy_unbook_va(void); +void *_ishmphy_map(int fd, void *start, uint64_t size, + int flags, int mmap_flags); +int _ishmphy_unmap(void *start, uint64_t len, int flags); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp_config_internal.h b/platform/linux-generic/include/odp_config_internal.h index 989ea08..3eabc7b 100644 --- a/platform/linux-generic/include/odp_config_internal.h +++ b/platform/linux-generic/include/odp_config_internal.h @@ -110,6 +110,16 @@ extern "C" { */ #define ODP_CONFIG_SHM_BLOCKS (ODP_CONFIG_POOLS + 48) +/* + * Size of the virtual address space pre-reserver for ISHM + * + * This is just virtual space preallocation size, not memory allocation. + * This address space is used by ISHM to map things at a common address in + * all ODP threads (when the _ODP_ISHM_SINGLE_VA flag is used). + * In bytes. + */ +#define ODP_CONFIG_ISHM_VA_PREALLOC_SZ (536870912L) + #ifdef __cplusplus } #endif diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index 2b21777..6ea8cf0 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -60,6 +60,7 @@ enum init_stage { TIME_INIT, SYSINFO_INIT, FDSERVER_INIT, + ISHM_INIT, SHM_INIT, THREAD_INIT, POOL_INIT, @@ -128,6 +129,11 @@ int _odp_int_name_tbl_term_global(void); int _odp_fdserver_init_global(void); int _odp_fdserver_term_global(void); +int _odp_ishm_init_global(void); +int _odp_ishm_init_local(void); +int _odp_ishm_term_global(void); +int _odp_ishm_term_local(void); + int cpuinfo_parser(FILE *file, system_info_t *sysinfo); uint64_t odp_cpu_hz_current(int id); diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index ec84fd0..d33a3ae 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -57,6 +57,12 @@ int odp_init_global(odp_instance_t *instance, } stage = FDSERVER_INIT; + if (_odp_ishm_init_global()) { + ODP_ERR("ODP ishm init failed.\n"); + goto init_failed; + } + stage = ISHM_INIT; + if (odp_shm_init_global()) { ODP_ERR("ODP shm init failed.\n"); goto init_failed; @@ -224,6 +230,13 @@ int _odp_term_global(enum init_stage stage) } /* Fall through */ + case ISHM_INIT: + if (_odp_ishm_term_global()) { + ODP_ERR("ODP ishm term failed.\n"); + rc = -1; + } + /* Fall through */ + case FDSERVER_INIT: if (_odp_fdserver_term_global()) { ODP_ERR("ODP fdserver term failed.\n"); @@ -268,6 +281,12 @@ int odp_init_local(odp_instance_t instance, odp_thread_type_t thr_type) goto init_fail; } + if (_odp_ishm_init_local()) { + ODP_ERR("ODP ishm local init failed.\n"); + goto init_fail; + } + stage = ISHM_INIT; + if (odp_shm_init_local()) { ODP_ERR("ODP shm local init failed.\n"); goto init_fail; @@ -343,6 +362,13 @@ int _odp_term_local(enum init_stage stage) } /* Fall through */ + case ISHM_INIT: + if (_odp_ishm_term_local()) { + ODP_ERR("ODP ishm local term failed.\n"); + rc = -1; + } + /* Fall through */ + default: break; } commit 442b93274d8f24f31557fe678245f65a2c499f97 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:45:57 2016 +0200 linux-generic: system_info: adding huge page dir The Huge page information is separated and a function to get the huge page mount directory is added. This function is called at init so the information is available later on. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index fd770b5..2b21777 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -30,7 +30,6 @@ extern __thread int __odp_errno; typedef struct { uint64_t cpu_hz_max[MAX_CPU_NUMBER]; - uint64_t default_huge_page_size; uint64_t page_size; int cache_line_size; int cpu_count; @@ -38,11 +37,17 @@ typedef struct { char model_str[MAX_CPU_NUMBER][128]; } system_info_t; +typedef struct { + uint64_t default_huge_page_size; + char *default_huge_page_dir; +} hugepage_info_t; + struct odp_global_data_s { pid_t main_pid; odp_log_func_t log_fn; odp_abort_func_t abort_fn; system_info_t system_info; + hugepage_info_t hugepage_info; odp_cpumask_t control_cpus; odp_cpumask_t worker_cpus; int num_cpus_installed; diff --git a/platform/linux-generic/odp_system_info.c b/platform/linux-generic/odp_system_info.c index bbe5358..18c61db 100644 --- a/platform/linux-generic/odp_system_info.c +++ b/platform/linux-generic/odp_system_info.c @@ -4,6 +4,13 @@ * SPDX-License-Identifier: BSD-3-Clause */ +/* + * BSD LICENSE + * + * Copyright(c) 2010-2014 Intel Corporation. All rights reserved. + * All rights reserved. + */ + #include <odp_posix_extensions.h> #include <odp/api/system_info.h> @@ -11,11 +18,13 @@ #include <odp_debug_internal.h> #include <odp/api/align.h> #include <odp/api/cpu.h> +#include <errno.h> #include <pthread.h> #include <sched.h> #include <string.h> #include <stdio.h> #include <inttypes.h> +#include <ctype.h> /* sysconf */ #include <unistd.h> @@ -97,6 +106,158 @@ static uint64_t default_huge_page_size(void) } /* + * split string into tokens. largely "inspired" by dpdk: + * lib/librte_eal/common/eal_common_string_fns.c: rte_strsplit + */ +static int strsplit(char *string, int stringlen, + char **tokens, int maxtokens, char delim) +{ + int i, tok = 0; + int tokstart = 1; /* first token is right at start of string */ + + if (string == NULL || tokens == NULL) + return -1; + + for (i = 0; i < stringlen; i++) { + if (string[i] == '\0' || tok >= maxtokens) + break; + if (tokstart) { + tokstart = 0; + tokens[tok++] = &string[i]; + } + if (string[i] == delim) { + string[i] = '\0'; + tokstart = 1; + } + } + return tok; +} + +/* + * Converts a numeric string to the equivalent uint64_t value. + * As well as straight number conversion, also recognises the suffixes + * k, m and g for kilobytes, megabytes and gigabytes respectively. + * + * If a negative number is passed in i.e. a string with the first non-black + * character being "-", zero is returned. Zero is also returned in the case of + * an error with the strtoull call in the function. + * largely "inspired" by dpdk: + * lib/librte_eal/common/include/rte_common.h: rte_str_to_size + * + * param str + * String containing number to convert. + * return + * Number. + */ +static inline uint64_t str_to_size(const char *str) +{ + char *endptr; + unsigned long long size; + + while (isspace((int)*str)) + str++; + if (*str == '-') + return 0; + + errno = 0; + size = strtoull(str, &endptr, 0); + if (errno) + return 0; + + if (*endptr == ' ') + endptr++; /* allow 1 space gap */ + + switch (*endptr) { + case 'G': + case 'g': + size *= 1024; /* fall-through */ + case 'M': + case 'm': + size *= 1024; /* fall-through */ + case 'K': + case 'k': + size *= 1024; /* fall-through */ + default: + break; + } + return size; +} + +/* + * returns a malloced string containing the name of the directory for + * huge pages of a given size (0 for default) + * largely "inspired" by dpdk: + * lib/librte_eal/linuxapp/eal/eal_hugepage_info.c: get_hugepage_dir + * + * Analysis of /proc/mounts + */ +static char *get_hugepage_dir(uint64_t hugepage_sz) +{ + enum proc_mount_fieldnames { + DEVICE = 0, + MOUNTPT, + FSTYPE, + OPTIONS, + _FIELDNAME_MAX + }; + static uint64_t default_size; + const char proc_mounts[] = "/proc/mounts"; + const char hugetlbfs_str[] = "hugetlbfs"; + const size_t htlbfs_str_len = sizeof(hugetlbfs_str) - 1; + const char pagesize_opt[] = "pagesize="; + const size_t pagesize_opt_len = sizeof(pagesize_opt) - 1; + const char split_tok = ' '; + char *tokens[_FIELDNAME_MAX]; + char buf[BUFSIZ]; + char *retval = NULL; + const char *pagesz_str; + uint64_t pagesz; + FILE *fd = fopen(proc_mounts, "r"); + + if (fd == NULL) + return NULL; + + if (default_size == 0) + default_size = default_huge_page_size(); + + if (hugepage_sz == 0) + hugepage_sz = default_size; + + while (fgets(buf, sizeof(buf), fd)) { + if (strsplit(buf, sizeof(buf), tokens, + _FIELDNAME_MAX, split_tok) != _FIELDNAME_MAX) { + ODP_ERR("Error parsing %s\n", proc_mounts); + break; /* return NULL */ + } + + /* is this hugetlbfs? */ + if (!strncmp(tokens[FSTYPE], hugetlbfs_str, htlbfs_str_len)) { + pagesz_str = strstr(tokens[OPTIONS], pagesize_opt); + + /* No explicit size, default page size is compared */ + if (pagesz_str == NULL) { + if (hugepage_sz == default_size) { + retval = strdup(tokens[MOUNTPT]); + break; + } + } + /* there is an explicit page size, so check it */ + else { + pagesz = + str_to_size(&pagesz_str[pagesize_opt_len]); + if (pagesz == hugepage_sz) { + retval = strdup(tokens[MOUNTPT]); + break; + } + } + } /* end if strncmp hugetlbfs */ + } /* end while fgets */ + + fclose(fd); + return retval; +} + +/* * Analysis of /sys/devices/system/cpu/ files */ static int systemcpu(system_info_t *sysinfo) @@ -125,11 +286,21 @@ static int systemcpu(system_info_t *sysinfo) return -1; } - sysinfo->default_huge_page_size = default_huge_page_size(); - return 0; } +/* + * Huge page information + */ +static int system_hp(hugepage_info_t *hugeinfo) +{ + hugeinfo->default_huge_page_size = default_huge_page_size(); + + /* default_huge_page_dir may be NULL if no huge page support */ + hugeinfo->default_huge_page_dir = get_hugepage_dir(0); + + return 0; +} /* * System info initialisation @@ -157,6 +328,8 @@ int odp_system_info_init(void) return -1; } + system_hp(&odp_global_data.hugepage_info); + return 0; } @@ -165,6 +338,8 @@ int odp_system_info_init(void) */ int odp_system_info_term(void) { + free(odp_global_data.hugepage_info.default_huge_page_dir); + return 0; } @@ -200,7 +375,7 @@ uint64_t odp_cpu_hz_max_id(int id) uint64_t odp_sys_huge_page_size(void) { - return odp_global_data.system_info.default_huge_page_size; + return odp_global_data.hugepage_info.default_huge_page_size; } uint64_t odp_sys_page_size(void) commit a4dd0bade1ca02eb91178834a159c9d26f140c38 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:45:56 2016 +0200 linux-gen: fdserver: new fdserver added A fdserver is added and started at init time. The role of the fdserver (file descriptor server) is to enable sharing of file descriptors between unrelated processes: processes which wish to share a file descriptor may register their files descriptor to the server and processes wishing to use the shared file descriptors can do a lookup. When registration occurs, a triple {context, key, fd} is provided to the server. The context identifies the client and scope (i.e. shmem). The key is implemented as a long, and can be whatever. The server won't care as long as keys are unique. The file descriptor can be retrieved by another process providing the same context and key. Files descriptors passed this way are converted on the fly during the unix domain socket communication that occurs between the server and its clients. This is done by using the ancillary control data part of the msg. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 271f5b5..cb7c8e9 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -112,6 +112,7 @@ odpdrvplatinclude_HEADERS = \ $(srcdir)/include/odp/drv/plat/strong_types.h noinst_HEADERS = \ + ${srcdir}/include/_fdserver_internal.h \ ${srcdir}/include/odp_align_internal.h \ ${srcdir}/include/odp_atomic_internal.h \ ${srcdir}/include/odp_buffer_inlines.h \ @@ -154,6 +155,7 @@ noinst_HEADERS = \ ${srcdir}/Makefile.inc __LIB__libodp_linux_la_SOURCES = \ + _fdserver.c \ odp_atomic.c \ odp_barrier.c \ odp_buffer.c \ diff --git a/platform/linux-generic/_fdserver.c b/platform/linux-generic/_fdserver.c new file mode 100644 index 0000000..bf36eb2 --- /dev/null +++ b/platform/linux-generic/_fdserver.c @@ -0,0 +1,655 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/* + * This file implements a file descriptor sharing server enabling + * sharing of file descriptors between processes, regardless of fork time. + * + * File descriptors are process scoped, but they can be "sent and converted + * on the fly" between processes using special unix domain socket ancillary + * data. + * The receiving process gets a file descriptor "pointing" to the same thing + * as the one sent (but the value of the file descriptor itself may be different + * from the one sent). + * Because ODP applications are responsible for creating ODP threads (i.e. + * pthreads or linux processes), ODP has no control on the order things happen: + * Nothing prevent a thread A to fork B and C, and then C creating a pktio + * which will be used by A and B to send/receive packets. + * Assuming this pktio uses a file descriptor, the latter will need to be + * shared between the processes, despite the "non convenient" fork time. + * The shared memory allocator is likely to use this as well to be able to + * share memory regardless of fork() time. + * This server handles a table of {(context,key)<-> fd} pair, and is + * interfaced by the following functions: + * + * _odp_fdserver_register_fd(context, key, fd_to_send); + * _odp_fdserver_deregister_fd(context, key); + * _odp_fdserver_lookup_fd(context, key); + * + * which are used to register/deregister or querry for file descriptor based + * on a context and key value couple, which has to be unique. + * + * Note again that the file descriptors stored here are local to this server + * process and get converted both when registered or looked up. + */ + +#include <odp_posix_extensions.h> +#include <odp/api/spinlock.h> +#include <odp_internal.h> +#include <odp_debug_internal.h> +#include <_fdserver_internal.h> + +#include <stdio.h> +#include <stdlib.h> +#include <errno.h> +#include <string.h> +#include <sys/types.h> +#include <signal.h> +#include <sys/socket.h> +#include <sys/un.h> +#include <unistd.h> +#include <inttypes.h> +#include <sys/mman.h> +#include <sys/wait.h> + +#define FDSERVER_SOCKPATH_MAXLEN 32 +#define FDSERVER_SOCKPATH_FORMAT "/tmp/odp-%d-fdserver" +#define FDSERVER_BACKLOG 5 + +#ifndef MAP_ANONYMOUS +#define MAP_ANONYMOUS MAP_ANON +#endif + +/* when accessing the client functions, clients should be mutexed: */ +odp_spinlock_t *client_lock; + +/* define the tables of file descriptors handled by this server: */ +#define FDSERVER_MAX_ENTRIES 256 +typedef struct fdentry_s { + fd_server_context_e context; + uint64_t key; + int fd; +} fdentry_t; +static fdentry_t fd_table[FDSERVER_MAX_ENTRIES]; +static int fd_table_nb_entries; + +/* + * define the message struct used for communication between client and server + * (this single message is used in both direction) + * The file descriptors are sent out of band as ancillary data for conversion. + */ +typedef struct fd_server_msg { + int command; + fd_server_context_e context; + uint64_t key; +} fdserver_msg_t; +/* possible commands are: */ +#define FD_REGISTER_REQ 1 /* client -> server */ +#define FD_REGISTER_ACK 2 /* server -> client */ +#define FD_REGISTER_NACK 3 /* server -> client */ +#define FD_LOOKUP_REQ 4 /* client -> server */ +#define FD_LOOKUP_ACK 5 /* server -> client */ +#define FD_LOOKUP_NACK 6 /* server -> client */ +#define FD_DEREGISTER_REQ 7 /* client -> server */ +#define FD_DEREGISTER_ACK 8 /* server -> client */ +#define FD_DEREGISTER_NACK 9 /* server -> client */ +#define FD_SERVERSTOP_REQ 10 /* client -> server (stops) */ + +/* + * Client and server function: + * Send a fdserver_msg, possibly including a file descriptor, on the socket + * This function is used both by: + * -the client (sending a FD_REGISTER_REQ with a file descriptor to be shared, + * or FD_LOOKUP_REQ/FD_DEREGISTER_REQ without a file descirptor) + * -the server (sending FD_REGISTER_ACK/NACK, FD_LOOKUP_NACK, + * FD_DEREGISTER_ACK/NACK... without a fd or a + * FD_LOOKUP_ACK with a fd) + * This function make use of the ancillary data (control data) to pass and + * convert file descriptors over UNIX sockets + * Return -1 on error, 0 on success. + */ +static int send_fdserver_msg(int sock, int command, + fd_server_context_e context, uint64_t key, + int fd_to_send) +{ + struct msghdr socket_message; + struct iovec io_vector[1]; /* one msg frgmt only */ + struct cmsghdr *control_message = NULL; + int *fd_location; + fdserver_msg_t msg; + int res; + + char ancillary_data[CMSG_SPACE(sizeof(int))]; + + /* prepare the register request body (single framgent): */ + msg.command = command; + msg.context = context; + msg.key = key; + io_vector[0].iov_base = &msg; + io_vector[0].iov_len = sizeof(fdserver_msg_t); + + /* initialize socket message */ + memset(&socket_message, 0, sizeof(struct msghdr)); + socket_message.msg_iov = io_vector; + socket_message.msg_iovlen = 1; + + if (fd_to_send >= 0) { + /* provide space for the ancillary data */ + memset(ancillary_data, 0, CMSG_SPACE(sizeof(int))); + socket_message.msg_control = ancillary_data; + socket_message.msg_controllen = CMSG_SPACE(sizeof(int)); + + /* initialize a single ancillary data element for fd passing */ + control_message = CMSG_FIRSTHDR(&socket_message); + control_message->cmsg_level = SOL_SOCKET; + control_message->cmsg_type = SCM_RIGHTS; + control_message->cmsg_len = CMSG_LEN(sizeof(int)); + fd_location = (int *)(void *)CMSG_DATA(control_message); + *fd_location = fd_to_send; + } + res = sendmsg(sock, &socket_message, 0); + if (res < 0) { + ODP_ERR("send_fdserver_msg: %s\n", strerror(errno)); + return(-1); + } + + return 0; +} + +/* + * Client and server function + * Receive a fdserver_msg, possibly including a file descriptor, on the + * given socket. + * This function is used both by: + * -the server (receiving a FD_REGISTER_REQ with a file descriptor to be shared, + * or FD_LOOKUP_REQ, FD_DEREGISTER_REQ without a file descirptor) + * -the client (receiving FD_REGISTER_ACK...without a fd or a FD_LOOKUP_ACK with + * a fd) + * This function make use of the ancillary data (control data) to pass and + * convert file descriptors over UNIX sockets. + * Return -1 on error, 0 on success. + */ +static int recv_fdserver_msg(int sock, int *command, + fd_server_context_e *context, uint64_t *key, + int *recvd_fd) +{ + struct msghdr socket_message; + struct iovec io_vector[1]; /* one msg frgmt only */ + struct cmsghdr *control_message = NULL; + int *fd_location; + fdserver_msg_t msg; + char ancillary_data[CMSG_SPACE(sizeof(int))]; + + memset(&socket_message, 0, sizeof(struct msghdr)); + memset(ancillary_data, 0, CMSG_SPACE(sizeof(int))); + + /* setup a place to fill in message contents */ + io_vector[0].iov_base = &msg; + io_vector[0].iov_len = sizeof(fdserver_msg_t); + socket_message.msg_iov = io_vector; + socket_message.msg_iovlen = 1; + + /* provide space for the ancillary data */ + socket_message.msg_control = ancillary_data; + socket_message.msg_controllen = CMSG_SPACE(sizeof(int)); + + /* receive the message */ + if (recvmsg(sock, &socket_message, MSG_CMSG_CLOEXEC) < 0) { + ODP_ERR("recv_fdserver_msg: %s\n", strerror(errno)); + return(-1); + } + + *command = msg.command; + *context = msg.context; + *key = msg.key; + + /* grab the converted file descriptor (if any) */ + *recvd_fd = -1; + + if ((socket_message.msg_flags & MSG_CTRUNC) == MSG_CTRUNC) + return 0; + + /* iterate ancillary elements to find the file descriptor: */ + for (control_message = CMSG_FIRSTHDR(&socket_message); + control_message != NULL; + control_message = CMSG_NXTHDR(&socket_message, control_message)) { + if ((control_message->cmsg_level == SOL_SOCKET) && + (control_message->cmsg_type == SCM_RIGHTS)) { + fd_location = (int *)(void *)CMSG_DATA(control_message); + *recvd_fd = *fd_location; + break; + } + } + + return 0; +} + +/* opens and returns a connected socket to the server */ +static int get_socket(void) +{ + char sockpath[FDSERVER_SOCKPATH_MAXLEN]; + int s_sock; /* server socket */ + struct sockaddr_un remote; + int len; + + /* construct the named socket path: */ + snprintf(sockpath, FDSERVER_SOCKPATH_MAXLEN, FDSERVER_SOCKPATH_FORMAT, + odp_global_data.main_pid); + + s_sock = socket(AF_UNIX, SOCK_STREAM, 0); + if (s_sock == -1) { + ODP_ERR("cannot connect to server: %s\n", strerror(errno)); + return(-1); + } + + remote.sun_family = AF_UNIX; + strcpy(remote.sun_path, sockpath); + len = strlen(remote.sun_path) + sizeof(remote.sun_family); + if (connect(s_sock, (struct sockaddr *)&remote, len) == -1) { + ODP_ERR("cannot connect to server: %s\n", strerror(errno)); + close(s_sock); + return(-1); + } + + return s_sock; +} + +/* + * Client function: + * Register a file descriptor to the server. Return -1 on error. + */ +int _odp_fdserver_register_fd(fd_server_context_e context, uint64_t key, + int fd_to_send) +{ + int s_sock; /* server socket */ + int res; + int command; + int fd; + + odp_spinlock_lock(client_lock); + + ODP_DBG("FD client register: pid=%d key=%" PRIu64 ", fd=%d\n", + getpid(), key, fd_to_send); + + s_sock = get_socket(); + if (s_sock < 0) { + odp_spinlock_unlock(client_lock); + return(-1); + } + + res = send_fdserver_msg(s_sock, FD_REGISTER_REQ, context, key, + fd_to_send); + if (res < 0) { + ODP_ERR("fd registration failure\n"); + close(s_sock); + odp_spinlock_unlock(client_lock); + return -1; + } + + res = recv_fdserver_msg(s_sock, &command, &context, &key, &fd); + + if ((res < 0) || (command != FD_REGISTER_ACK)) { + ODP_ERR("fd registration failure\n"); + close(s_sock); + odp_spinlock_unlock(client_lock); + return -1; + } + + close(s_sock); + + odp_spinlock_unlock(client_lock); + return 0; +} + +/* + * Client function: + * Deregister a file descriptor from the server. Return -1 on error. + */ +int _odp_fdserver_deregister_fd(fd_server_context_e context, uint64_t key) +{ + int s_sock; /* server socket */ + int res; + int command; + int fd; + + odp_spinlock_lock(client_lock); + + ODP_DBG("FD client deregister: pid=%d key=%" PRIu64 "\n", + getpid(), key); + + s_sock = get_socket(); + if (s_sock < 0) { + odp_spinlock_unlock(client_lock); + return(-1); + } + + res = send_fdserver_msg(s_sock, FD_DEREGISTER_REQ, context, key, -1); + if (res < 0) { + ODP_ERR("fd de-registration failure\n"); + close(s_sock); + odp_spinlock_unlock(client_lock); + return -1; + } + + res = recv_fdserver_msg(s_sock, &command, &context, &key, &fd); + + if ((res < 0) || (command != FD_DEREGISTER_ACK)) { + ODP_ERR("fd de-registration failure\n"); + close(s_sock); + odp_spinlock_unlock(client_lock); + return -1; + } + + close(s_sock); + + odp_spinlock_unlock(client_lock); + return 0; +} + +/* + * client function: + * lookup a file descriptor from the server. return -1 on error, + * or the file descriptor on success (>=0). + */ +int _odp_fdserver_lookup_fd(fd_server_context_e context, uint64_t key) +{ + int s_sock; /* server socket */ + int res; + int command; + int fd; + + odp_spinlock_lock(client_lock); + + s_sock = get_socket(); + if (s_sock < 0) { + odp_spinlock_unlock(client_lock); + return(-1); + } + + res = send_fdserver_msg(s_sock, FD_LOOKUP_REQ, context, key, -1); + if (res < 0) { + ODP_ERR("fd lookup failure\n"); + close(s_sock); + odp_spinlock_unlock(client_lock); + return -1; + } + + res = recv_fdserver_msg(s_sock, &command, &context, &key, &fd); + + if ((res < 0) || (command != FD_LOOKUP_ACK)) { + ODP_ERR("fd lookup failure\n"); + close(s_sock); + odp_spinlock_unlock(client_lock); + return -1; + } + + close(s_sock); + ODP_DBG("FD client lookup: pid=%d, key=%" PRIu64 ", fd=%d\n", + getpid(), key, fd); + + odp_spinlock_unlock(client_lock); + return fd; +} + +/* + * request server terminaison: + */ +static int stop_server(void) +{ + int s_sock; /* server socket */ + int res; + + odp_spinlock_lock(client_lock); + + ODP_DBG("FD sending server stop request\n"); + + s_sock = get_socket(); + if (s_sock < 0) { + odp_spinlock_unlock(client_lock); + return(-1); + } + + res = send_fdserver_msg(s_sock, FD_SERVERSTOP_REQ, 0, 0, -1); + if (res < 0) { + ODP_ERR("fd stop request failure\n"); + close(s_sock); + odp_spinlock_unlock(client_lock); + return -1; + } + + close(s_sock); + + odp_spinlock_unlock(client_lock); + return 0; +} + +/* + * server function + * receive a client request and handle it. + * Always returns 0 unless a stop request is received. + */ +static int handle_request(int client_sock) +{ + int command; + fd_server_context_e context; + uint64_t key; + int fd; + int i; + + /* get a client request: */ + recv_fdserver_msg(client_sock, &command, &context, &key, &fd); + switch (command) { + case FD_REGISTER_REQ: + if ((fd < 0) || (context >= FD_SRV_CTX_END)) { + ODP_ERR("Invalid register fd or context\n"); + send_fdserver_msg(client_sock, FD_REGISTER_NACK, + FD_SRV_CTX_NA, 0, -1); + return 0; + } + + /* store the file descriptor in table: */ + if (fd_table_nb_entries < FDSERVER_MAX_ENTRIES) { + fd_table[fd_table_nb_entries].context = context; + fd_table[fd_table_nb_entries].key = key; + fd_table[fd_table_nb_entries++].fd = fd; + ODP_DBG("storing {ctx=%d, key=%" PRIu64 "}->fd=%d\n", + context, key, fd); + } else { + ODP_ERR("FD table full\n"); + send_fdserver_msg(client_sock, FD_REGISTER_NACK, + FD_SRV_CTX_NA, 0, -1); + return 0; + } + + send_fdserver_msg(client_sock, FD_REGISTER_ACK, + FD_SRV_CTX_NA, 0, -1); + break; + + case FD_LOOKUP_REQ: + if (context >= FD_SRV_CTX_END) { + ODP_ERR("invalid lookup context\n"); + send_fdserver_msg(client_sock, FD_LOOKUP_NACK, + FD_SRV_CTX_NA, 0, -1); + return 0; + } + + /* search key in table and sent reply: */ + for (i = 0; i < fd_table_nb_entries; i++) { + if ((fd_table[i].context == context) && + (fd_table[i].key == key)) { + fd = fd_table[i].fd; + ODP_DBG("lookup {ctx=%d," + " key=%" PRIu64 "}->fd=%d\n", + context, key, fd); + send_fdserver_msg(client_sock, + FD_LOOKUP_ACK, context, key, + fd); + return 0; + } + } + + /* context+key not found... send nack */ + send_fdserver_msg(client_sock, FD_LOOKUP_NACK, context, key, + -1); + break; + + case FD_DEREGISTER_REQ: + if (context >= FD_SRV_CTX_END) { + ODP_ERR("invalid deregister context\n"); + send_fdserver_msg(client_sock, FD_DEREGISTER_NACK, + FD_SRV_CTX_NA, 0, -1); + return 0; + } + + /* search key in table and remove it if found, and reply: */ + for (i = 0; i < fd_table_nb_entries; i++) { + if ((fd_table[i].context == context) && + (fd_table[i].key == key)) { + ODP_DBG("drop {ctx=%d," + " key=%" PRIu64 "}->fd=%d\n", + context, key, fd_table[i].fd); + close(fd_table[i].fd); + fd_table[i] = fd_table[--fd_table_nb_entries]; + send_fdserver_msg(client_sock, + FD_DEREGISTER_ACK, + context, key, -1); + return 0; + } + } + + /* key not found... send nack */ + send_fdserver_msg(client_sock, FD_DEREGISTER_NACK, + context, key, -1); + break; + + case FD_SERVERSTOP_REQ: + ODP_DBG("Stoping FD server\n"); + return 1; + + default: + ODP_ERR("Unexpected request\n"); + break; + } + return 0; +} + +/* + * server function + * loop forever, handling client requests one by one + */ +static void wait_requests(int sock) +{ + int c_socket; /* client connection */ + unsigned int addr_sz; + struct sockaddr_un remote; + + for (;;) { + addr_sz = sizeof(remote); + c_socket = accept(sock, (struct sockaddr *)&remote, &addr_sz); + if (c_socket == -1) { + ODP_ERR("wait_requests: %s\n", strerror(errno)); + return; + } + + if (handle_request(c_socket)) + break; + close(c_socket); + } + close(c_socket); +} + +/* + * Create a unix domain socket and fork a process to listen to incoming + * requests. + */ +int _odp_fdserver_init_global(void) +{ + char sockpath[FDSERVER_SOCKPATH_MAXLEN]; + int sock; + struct sockaddr_un local; + pid_t server_pid; + int res; + + /* create the client spinlock that any client can see: */ + client_lock = mmap(NULL, sizeof(odp_spinlock_t), PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_ANONYMOUS, -1, 0); + + odp_spinlock_init(client_lock); + + /* construct the server named socket path: */ + snprintf(sockpath, FDSERVER_SOCKPATH_MAXLEN, FDSERVER_SOCKPATH_FORMAT, + odp_global_data.main_pid); + + /* create UNIX domain socket: */ + sock = socket(AF_UNIX, SOCK_STREAM, 0); + if (sock == -1) { + ODP_ERR("_odp_fdserver_init_global: %s\n", strerror(errno)); + return(-1); + } + + /* remove previous named socket if it already exists: */ + unlink(sockpath); + + /* bind to new named socket: */ + local.sun_family = AF_UNIX; + strncpy(local.sun_path, sockpath, sizeof(local.sun_path)); + res = bind(sock, (struct sockaddr *)&local, sizeof(struct sockaddr_un)); + if (res == -1) { + ODP_ERR("_odp_fdserver_init_global: %s\n", strerror(errno)); + close(sock); + return(-1); + } + + /* listen for incoming conections: */ + if (listen(sock, FDSERVER_BACKLOG) == -1) { + ODP_ERR("_odp_fdserver_init_global: %s\n", strerror(errno)); + close(sock); + return(-1); + } + + /* fork a server process: */ + server_pid = fork(); + if (server_pid == -1) { + ODP_ERR("Could not fork!\n"); + close(sock); + return(-1); + } + + if (server_pid == 0) { /*child */ + /* TODO: pin the server on appropriate service cpu mask */ + /* when (if) we can agree on the usage of service mask */ + + wait_requests(sock); /* Returns when server is stopped */ + close(sock); + exit(0); + } + + /* parent */ + close(sock); + return 0; +} + +/* + * Terminate the server + */ +int _odp_fdserver_term_global(void) +{ + int status; + char sockpath[FDSERVER_SOCKPATH_MAXLEN]; + + /* close the server and wait for child terminaison*/ + stop_server(); + wait(&status); + + /* construct the server named socket path: */ + snprintf(sockpath, FDSERVER_SOCKPATH_MAXLEN, FDSERVER_SOCKPATH_FORMAT, + odp_global_data.main_pid); + + /* delete the UNIX domain socket: */ + unlink(sockpath); + + return 0; +} diff --git a/platform/linux-generic/include/_fdserver_internal.h b/platform/linux-generic/include/_fdserver_internal.h new file mode 100644 index 0000000..480ac02 --- /dev/null +++ b/platform/linux-generic/include/_fdserver_internal.h @@ -0,0 +1,38 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _FD_SERVER_INTERNAL_H +#define _FD_SERVER_INTERNAL_H + +#ifdef __cplusplus +extern "C" { +#endif + +/* + * the following enum defines the different contextes by which the + * FD server may be used: In the FD server, the keys used to store/retrieve + * a file descriptor are actually context based: + * Both the context and the key are stored at fd registration time, + * and both the context and the key are used to retrieve a fd. + * In other words a context identifies a FD server usage, so that different + * unrelated fd server users do not have to guarantee key unicity between + * them. + */ +typedef enum fd_server_context { + FD_SRV_CTX_NA, /* Not Applicable */ + FD_SRV_CTX_END, /* upper enum limit */ +} fd_server_context_e; + +int _odp_fdserver_register_fd(fd_server_context_e context, uint64_t key, + int fd); +int _odp_fdserver_deregister_fd(fd_server_context_e context, uint64_t key); +int _odp_fdserver_lookup_fd(fd_server_context_e context, uint64_t key); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index 8bad450..fd770b5 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -54,6 +54,7 @@ enum init_stage { CPUMASK_INIT, TIME_INIT, SYSINFO_INIT, + FDSERVER_INIT, SHM_INIT, THREAD_INIT, POOL_INIT, @@ -119,6 +120,9 @@ int odp_tm_term_global(void); int _odp_int_name_tbl_init_global(void); int _odp_int_name_tbl_term_global(void); +int _odp_fdserver_init_global(void); +int _odp_fdserver_term_global(void); + int cpuinfo_parser(FILE *file, system_info_t *sysinfo); uint64_t odp_cpu_hz_current(int id); diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index f534759..ec84fd0 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -51,6 +51,12 @@ int odp_init_global(odp_instance_t *instance, } stage = SYSINFO_INIT; + if (_odp_fdserver_init_global()) { + ODP_ERR("ODP fdserver init failed.\n"); + goto init_failed; + } + stage = FDSERVER_INIT; + if (odp_shm_init_global()) { ODP_ERR("ODP shm init failed.\n"); goto init_failed; @@ -218,6 +224,13 @@ int _odp_term_global(enum init_stage stage) } /* Fall through */ + case FDSERVER_INIT: + if (_odp_fdserver_term_global()) { + ODP_ERR("ODP fdserver term failed.\n"); + rc = -1; + } + /* Fall through */ + case SYSINFO_INIT: if (odp_system_info_term()) { ODP_ERR("ODP system info term failed.\n"); commit 4871852cc509ca998802616fadfbb5619e3324da Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:45:55 2016 +0200 drv: adding shared memory The public definition of the shared memory allocator for the driver interface is added. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/include/odp/drv/spec/shm.h b/include/odp/drv/spec/shm.h new file mode 100644 index 0000000..ef64f5d --- /dev/null +++ b/include/odp/drv/spec/shm.h @@ -0,0 +1,231 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODPRDV shared memory (shm) + */ + +#ifndef ODPDRV_SHM_H_ +#define ODPDRV_SHM_H_ +#include <odp/visibility_begin.h> + +#ifdef __cplusplus +extern "C" { +#endif + +/** @addtogroup odpdrv_shm ODPDRV SHARED MEMORY + * Operations on driver shared memory. + * @{ + */ + +/** + * @typedef odpdrv_shm_t + * odpdrv shared memory block + */ + +/** + * @def ODPDRV_SHM_INVALID + * Invalid odpdrv shared memory block + */ + +/** Maximum shared memory block name length in chars */ +#define ODPDRV_SHM_NAME_LEN 32 + +/* + * Shared memory flags + */ +#define ODPDRV_SHM_SINGLE_VA 0x01 /**< Memory shall be mapped at same VA */ +#define ODPDRV_SHM_LOCK 0x02 /**< Memory shall be locked (no swap) */ + +/** + * Shared memory block info + */ +typedef struct odpdrv_shm_info_t { + const char *name; /**< Block name */ + void *addr; /**< Block address */ + uint64_t size; /**< Block size in bytes */ + uint64_t page_size; /**< Memory page size */ + uint32_t flags; /**< ODPDRV_SHM_* flags */ +} odpdrv_shm_info_t; + +/** + * Shared memory capabilities + */ +typedef struct odpdrv_shm_capability_t { + /** Maximum number of shared memory blocks + * + * This number of separate shared memory blocks can be + * reserved concurrently. */ + unsigned max_blocks; + + /** Maximum memory block size in bytes + * + * The value of zero means that size is limited only by the available + * memory size. */ + uint64_t max_size; + + /** Maximum memory block alignment in bytes + * + * The value of zero means that alignment is limited only by the + * available memory size. */ + uint64_t max_align; + +} odpdrv_shm_capability_t; + +/** + * Query shared memory capabilities + * + * Outputs shared memory capabilities on success. + * + * @param[out] capa Pointer to capability structure for output + * + * @retval 0 on success + * @retval <0 on failure + */ +int odpdrv_shm_capability(odpdrv_shm_capability_t *capa); + +/** + * Reserve a contiguous block of shared memory + * + * @param[in] name Name of the block (maximum ODPDRV_SHM_NAME_LEN - 1 chars) + * @param[in] size Block size in bytes + * @param[in] align Block alignment in bytes + * @param[in] flags Shared memory parameter flags (ODPDRV_SHM_*). + * Default value is 0. + * + * @return Handle of the reserved block + * @retval ODPDRV_SHM_INVALID on failure + */ +odpdrv_shm_t odpdrv_shm_reserve(const char *name, uint64_t size, uint64_t align, + uint32_t flags); + +/** + * Free a contiguous block of shared memory + * + * Frees a previously reserved block of shared memory (found by its handle). + * @note Freeing memory that is in use will result in UNDEFINED behavior + * + * @param[in] shm odpdrv_shm Block handle + * + * @retval 0 on success + * @retval <0 on failure + */ +int odpdrv_shm_free_by_handle(odpdrv_shm_t shm); + +/** + * Free a contiguous block of shared memory (found from its name) + * + * Frees a previously reserved block of shared memory. + * @note Freeing memory that is in use will result in UNDEFINED behavior + * + * @param[in] name odpdrv_shm Block name + * + * @retval 0 on success + * @retval <0 on failure + */ +int odpdrv_shm_free_by_name(const char *name); + +/** + * Free a contiguous block of shared memory (found from its address) + * + * Frees a previously reserved block of shared memory. + * @note Freeing memory that is in use will result in UNDEFINED behavior + * + * @param[in] address odpdrv_shm Block address + * + * @retval 0 on success + * @retval <0 on failure + */ +int odpdrv_shm_free_by_address(void *address); + +/** + * Lookup and map a block of shared memory (identified by its handle) + * + * @param[in] shm odpdrv_shm Block handle + * + * @return The address of the newly mapped block. + * @retval NULL on failure + */ +void *odpdrv_shm_lookup_by_handle(odpdrv_shm_t shm); + +/** + * Lookup and map a block of shared memory (identified by its name) + * + * @param[in] name odpdrv_shm Block name + * + * @return The handle of the newly mapped block. + * @retval ODPDRV_SHM_INVALID on failure + */ +odpdrv_shm_t odpdrv_shm_lookup_by_name(const char *name); + +/** + * Lookup and map a block of shared memory (identified by its address) + * + * @note This only works when the flag ODPDRV_SHM_SINGLE_VA was set, + * as otherwise addresses are odp-thread local and hence meaningless to + * identify the block between odp-threads. + * + * @param[in] address odpdrv_shm Block address + * + * @return The handle of the newly mapped block. + * @retval ODPDRV_SHM_INVALID on failure + */ +odpdrv_shm_t odpdrv_shm_lookup_by_address(void *address); + +/** + * Get a Shared memory block address + * + * @param[in] shm odpdrv_shm Block handle + * + * @return Memory block address + * @retval NULL on failure + */ +void *odpdrv_shm_addr(odpdrv_shm_t shm); + +/** + * Shared memory block info + * + * @param[in] shm Odpdrv_shm block handle + * @param[out] info Block info pointer for output + * + * @retval 0 on success + * @retval <0 on failure + */ +int odpdrv_shm_info(odpdrv_shm_t shm, odpdrv_shm_info_t *info); + +/** + * Print all shared memory blocks and returns the number of allocated blocks. + * This function is meant for debug. + * @param title A string to be printed before the shared memory status + * @return The total number of allocated blocks + */ +int odpdrv_shm_print_all(const char *title); + +/** + * Get printable value for an odpdrv_shm_t + * + * @param hdl odpdrv_shm_t handle to be printed + * @return uint64_t value that can be used to print/display this + * handle + * + * @note This routine is intended to be used for diagnostic purposes + * to enable applications to generate a printable value that represents + * an odpdrv_shm_t handle. + */ +uint64_t odpdrv_shm_to_u64(odpdrv_shm_t hdl); + +/** + * @} + */ + +#ifdef __cplusplus +} +#endif + +#include <odp/visibility_end.h> +#endif diff --git a/platform/Makefile.inc b/platform/Makefile.inc index 6ade6df..7c53d1d 100644 --- a/platform/Makefile.inc +++ b/platform/Makefile.inc @@ -68,6 +68,7 @@ odpdrvspecinclude_HEADERS = \ $(top_srcdir)/include/odp/drv/spec/barrier.h \ $(top_srcdir)/include/odp/drv/spec/byteorder.h \ $(top_srcdir)/include/odp/drv/spec/compiler.h \ + $(top_srcdir)/include/odp/drv/spec/shm.h \ $(top_srcdir)/include/odp/drv/spec/spinlock.h \ $(top_srcdir)/include/odp/drv/spec/std_types.h \ $(top_srcdir)/include/odp/drv/spec/sync.h commit 48ae68961873bb818a75aa8aa1b93d697fe39517 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:45:54 2016 +0200 linux-generic: drv: strong typing Mirrored from its couterpart on the api interface, the drv interface gets its strong typing file. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 42e9996..271f5b5 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -108,7 +108,8 @@ odpdrvplatinclude_HEADERS = \ $(srcdir)/include/odp/drv/plat/atomic_types.h \ $(srcdir)/include/odp/drv/plat/barrier_types.h \ $(srcdir)/include/odp/drv/plat/byteorder_types.h \ - $(srcdir)/include/odp/drv/plat/spinlock_types.h + $(srcdir)/include/odp/drv/plat/spinlock_types.h \ + $(srcdir)/include/odp/drv/plat/strong_types.h noinst_HEADERS = \ ${srcdir}/include/odp_align_internal.h \ diff --git a/platform/linux-generic/include/odp/drv/plat/strong_types.h b/platform/linux-generic/include/odp/drv/plat/strong_types.h new file mode 100644 index 0000000..a9c48ef --- /dev/null +++ b/platform/linux-generic/include/odp/drv/plat/strong_types.h @@ -0,0 +1,35 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODPDRV Strong Types. Common macros for implementing strong typing + * for ODPDRV abstract data types + */ + +#ifndef ODPDRV_STRONG_TYPES_H_ +#define ODPDRV_STRONG_TYPES_H_ + +/** Use strong typing for ODPDRV types */ +#ifdef __cplusplus +#define ODPDRV_HANDLE_T(type) struct _##type { uint8_t unused_dummy_var; } *type +#else +#define odpdrv_handle_t struct { uint8_t unused_dummy_var; } * +/** C/C++ helper macro for strong typing */ +#define ODPDRV_HANDLE_T(type) odpdrv_handle_t type +#endif + +/** Internal macro to get value of an ODPDRV handle */ +#define _odpdrv_typeval(handle) ((uint32_t)(uintptr_t)(handle)) + +/** Internal macro to get printable value of an ODPDRV handle */ +#define _odpdrv_pri(handle) ((uint64_t)_odpdrv_typeval(handle)) + +/** Internal macro to convert a scalar to a typed handle */ +#define _odpdrv_cast_scalar(type, val) ((type)(uintptr_t)(val)) + +#endif commit 85c4f0bd3f18900e484b0e71670fc18c555e75b1 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:45:53 2016 +0200 linux-gen: drv: adding barrier Based on API interface files. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/include/odp_drv.h b/include/odp_drv.h index 776cf12..31c620e 100644 --- a/include/odp_drv.h +++ b/include/odp_drv.h @@ -20,6 +20,7 @@ extern C { #include <odp/drv/align.h> #include <odp/drv/atomic.h> +#include <odp/drv/barrier.h> #include <odp/drv/byteorder.h> #include <odp/drv/compiler.h> #include <odp/drv/spinlock.h> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 747e96f..42e9996 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -96,6 +96,7 @@ odpdrvincludedir = $(includedir)/odp/drv odpdrvinclude_HEADERS = \ $(srcdir)/include/odp/drv/align.h \ $(srcdir)/include/odp/drv/atomic.h \ + $(srcdir)/include/odp/drv/barrier.h \ $(srcdir)/include/odp/drv/byteorder.h \ $(srcdir)/include/odp/drv/compiler.h \ $(srcdir)/include/odp/drv/spinlock.h \ @@ -105,6 +106,7 @@ odpdrvinclude_HEADERS = \ odpdrvplatincludedir = $(includedir)/odp/drv/plat odpdrvplatinclude_HEADERS = \ $(srcdir)/include/odp/drv/plat/atomic_types.h \ + $(srcdir)/include/odp/drv/plat/barrier_types.h \ $(srcdir)/include/odp/drv/plat/byteorder_types.h \ $(srcdir)/include/odp/drv/plat/spinlock_types.h @@ -204,6 +206,7 @@ __LIB__libodp_linux_la_SOURCES = \ odp_version.c \ odp_weak.c \ drv_atomic.c \ + drv_barrier.c \ drv_spinlock.c \ arch/@ARCH_DIR@/odp_cpu_arch.c \ arch/@ARCH_DIR@/odp_sysinfo_parse.c diff --git a/platform/linux-generic/drv_barrier.c b/platform/linux-generic/drv_barrier.c new file mode 100644 index 0000000..7a83981 --- /dev/null +++ b/platform/linux-generic/drv_barrier.c @@ -0,0 +1,50 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp/drv/barrier.h> +#include <odp/drv/sync.h> +#include <odp/api/cpu.h> +#include <odp/drv/atomic.h> + +void odpdrv_barrier_init(odpdrv_barrier_t *barrier, int count) +{ + barrier->count = (uint32_t)count; + odpdrv_atomic_init_u32(&barrier->bar, 0); +} + +/* + * Efficient barrier_sync - + * + * Barriers are initialized with a count of the number of callers + * that must sync on the barrier before any may proceed. + * + * To avoid race conditions and to permit the barrier to be fully + * reusable, the barrier value cycles between 0..2*count-1. When + * synchronizing the wasless variable simply tracks which half of + * the cycle the barrier was in upon entry. Exit is when the + * barrier crosses to the other half of the cycle. + */ +void odpdrv_barrier_wait(odpdrv_barrier_t *barrier) +{ + uint32_t count; + int wasless; + + odpdrv_mb_full(); + + count = odpdrv_atomic_fetch_inc_u32(&barrier->bar); + wasless = count < barrier->count; + + if (count == 2 * barrier->count - 1) { + /* Wrap around *atomically* */ + odpdrv_atomic_sub_u32(&barrier->bar, 2 * barrier->count); + } else { + while ((odpdrv_atomic_load_u32(&barrier->bar) < barrier->count) + == wasless) + odp_cpu_pause(); + } + + odpdrv_mb_full(); +} diff --git a/platform/linux-generic/include/odp/drv/barrier.h b/platform/linux-generic/include/odp/drv/barrier.h new file mode 100644 index 0000000..877cacf --- /dev/null +++ b/platform/linux-generic/include/odp/drv/barrier.h @@ -0,0 +1,30 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODPDRV execution barriers + */ + +#ifndef ODPDRV_PLAT_BARRIER_H_ +#define ODPDRV_PLAT_BARRIER_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp/drv/std_types.h> +#include <odp/drv/atomic.h> +#include <odp/drv/plat/barrier_types.h> + +#include <odp/drv/spec/barrier.h> + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp/drv/plat/barrier_types.h b/platform/linux-generic/include/odp/drv/plat/barrier_types.h new file mode 100644 index 0000000..32264a1 --- /dev/null +++ b/platform/linux-generic/include/odp/drv/plat/barrier_types.h @@ -0,0 +1,38 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODPDRV barrier + */ + +#ifndef ODPDRV_BARRIER_TYPES_H_ +#define ODPDRV_BARRIER_TYPES_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp/drv/std_types.h> +#include <odp/drv/atomic.h> + +/** + * @internal + * ODPDRV thread synchronization barrier + */ +struct odpdrv_barrier_s { + uint32_t count; /**< Thread count */ + odpdrv_atomic_u32_t bar; /**< Barrier counter */ +}; + +typedef struct odpdrv_barrier_s odpdrv_barrier_t; + +#ifdef __cplusplus +} +#endif + +#endif commit 57962f1b9c3a5a41a7dd6146f9666da18e62ec9c Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:45:52 2016 +0200 drv: adding barrier.h Based on API interface file. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/include/odp/drv/spec/barrier.h b/include/odp/drv/spec/barrier.h new file mode 100644 index 0000000..5ac65a3 --- /dev/null +++ b/include/odp/drv/spec/barrier.h @@ -0,0 +1,66 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODPDRV execution barriers + */ + +#ifndef ODPDRV_API_BARRIER_H_ +#define ODPDRV_API_BARRIER_H_ +#include <odp/visibility_begin.h> + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * @defgroup odpdrv_barrier ODPDRV BARRIER + * Thread execution and memory ordering barriers. + * + * @details + * <b> Thread execution barrier (odpdrv_barrier_t) </b> + * + * Thread execution barrier synchronizes a group of threads to wait on the + * barrier until the entire group has reached the barrier. + * @{ + */ + +/** + * @typedef odpdrv_barrier_t + * ODPDRV thread synchronization barrier + */ + +/** + * Initialize barrier with thread count. + * + * @param barr Pointer to a barrier variable + * @param count Thread count + */ +void odpdrv_barrier_init(odpdrv_barrier_t *barr, int count); + +/** + * Synchronize thread execution on barrier. + * Wait for all threads to arrive at the barrier until they are let loose again. + * Threads will block (spin) until the last thread has arrived at the barrier. + * All memory operations before the odpdrv_barrier_wait() call will be visible + * to all threads when they leave the barrier. + * + * @param barr Pointer to a barrier variable + */ +void odpdrv_barrier_wait(odpdrv_barrier_t *barr); + +/** + * @} + */ + +#ifdef __cplusplus +} +#endif + +#include <odp/visibility_end.h> +#endif diff --git a/platform/Makefile.inc b/platform/Makefile.inc index 079f34d..6ade6df 100644 --- a/platform/Makefile.inc +++ b/platform/Makefile.inc @@ -65,6 +65,7 @@ odpdrvspecincludedir= $(includedir)/odp/drv/spec odpdrvspecinclude_HEADERS = \ $(top_srcdir)/include/odp/drv/spec/align.h \ $(top_srcdir)/include/odp/drv/spec/atomic.h \ + $(top_srcdir)/include/odp/drv/spec/barrier.h \ $(top_srcdir)/include/odp/drv/spec/byteorder.h \ $(top_srcdir)/include/odp/drv/spec/compiler.h \ $(top_srcdir)/include/odp/drv/spec/spinlock.h \ commit 9a45802c94f58d9dd443c10e6dc204a58fa0f9ce Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Sat Aug 20 09:45:51 2016 +0200 linux-gen: cosmetic changes on barrier To please check-patch before the copy to the drv interface. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Brian Brooks <brian.brooks(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/include/odp/api/spec/barrier.h b/include/odp/api/spec/barrier.h index 678d39a..6de683c 100644 --- a/include/odp/api/spec/barrier.h +++ b/include/odp/api/spec/barrier.h @@ -4,7 +4,6 @@ * SPDX-License-Identifier: BSD-3-Clause */ - /** * @file * @@ -44,7 +43,6 @@ extern "C" { */ void odp_barrier_init(odp_barrier_t *barr, int count); - /** * Synchronize thread execution on barrier. * Wait for all threads to arrive at the barrier until they are let loose again. diff --git a/platform/linux-generic/include/odp/api/plat/barrier_types.h b/platform/linux-generic/include/odp/api/plat/barrier_types.h index 440275e..00b383c 100644 --- a/platform/linux-generic/include/odp/api/plat/barrier_types.h +++ b/platform/linux-generic/include/odp/api/plat/barrier_types.h @@ -4,7 +4,6 @@ * SPDX-License-Identifier: BSD-3-Clause */ - /** * @file * diff --git a/platform/linux-generic/odp_barrier.c b/platform/linux-generic/odp_barrier.c index ef10f29..a2c6267 100644 --- a/platform/linux-generic/odp_barrier.c +++ b/platform/linux-generic/odp_barrier.c @@ -37,7 +37,7 @@ void odp_barrier_wait(odp_barrier_t *barrier) count = odp_atomic_fetch_inc_u32(&barrier->bar); wasless = count < barrier->count; - if (count == 2*barrier->count-1) { + if (count == 2 * barrier->count - 1) { /* Wrap around *atomically* */ odp_atomic_sub_u32(&barrier->bar, 2 * barrier->count); } else { ----------------------------------------------------------------------- Summary of changes: include/odp/api/spec/barrier.h | 2 - include/odp/{api => drv}/spec/barrier.h | 24 +- include/odp/drv/spec/shm.h | 231 ++++ include/odp_drv.h | 2 + platform/Makefile.inc | 2 + platform/linux-generic/Makefile.am | 15 +- platform/linux-generic/_fdserver.c | 655 +++++++++ platform/linux-generic/_ishm.c | 1460 ++++++++++++++++++++ platform/linux-generic/_ishmphy.c | 185 +++ platform/linux-generic/arch/arm/odp/api/cpu_arch.h | 25 +- platform/linux-generic/arch/arm/odp_cpu_arch.c | 49 +- .../linux-generic/arch/arm/odp_sysinfo_parse.c | 28 +- platform/linux-generic/arch/powerpc/odp_cpu_arch.c | 49 +- .../linux-generic/{odp_barrier.c => drv_barrier.c} | 26 +- platform/linux-generic/drv_shm.c | 102 ++ .../linux-generic/include/_fdserver_internal.h | 39 + platform/linux-generic/include/_ishm_internal.h | 46 + platform/linux-generic/include/_ishmphy_internal.h | 25 + platform/linux-generic/include/ishmphy_internal.h | 24 + .../include/odp/api/plat/barrier_types.h | 1 - platform/linux-generic/include/odp/drv/barrier.h | 30 + .../include/odp/drv/plat/barrier_types.h | 38 + .../linux-generic/include/odp/drv/plat/shm_types.h | 46 + .../include/odp/drv/plat/strong_types.h | 35 + .../include/odp/drv/{compiler.h => shm.h} | 12 +- .../linux-generic/include/odp_config_internal.h | 16 + platform/linux-generic/include/odp_internal.h | 17 +- platform/linux-generic/odp_barrier.c | 2 +- platform/linux-generic/odp_init.c | 39 + platform/linux-generic/odp_system_info.c | 181 ++- test/common_plat/m4/configure.m4 | 3 +- test/common_plat/validation/drv/Makefile.am | 3 +- .../common_plat/validation/drv/drvshmem/.gitignore | 1 + .../validation/drv/drvshmem/Makefile.am | 10 + .../common_plat/validation/drv/drvshmem/drvshmem.c | 794 +++++++++++ .../common_plat/validation/drv/drvshmem/drvshmem.h | 27 + .../drvatomic_main.c => drvshmem/drvshmem_main.c} | 4 +- test/linux-generic/Makefile.am | 1 + 38 files changed, 4201 insertions(+), 48 deletions(-) copy include/odp/{api => drv}/spec/barrier.h (65%) create mode 100644 include/odp/drv/spec/shm.h create mode 100644 platform/linux-generic/_fdserver.c create mode 100644 platform/linux-generic/_ishm.c create mode 100644 platform/linux-generic/_ishmphy.c mode change 120000 => 100644 platform/linux-generic/arch/arm/odp/api/cpu_arch.h mode change 120000 => 100644 platform/linux-generic/arch/arm/odp_cpu_arch.c mode change 120000 => 100644 platform/linux-generic/arch/arm/odp_sysinfo_parse.c mode change 120000 => 100644 platform/linux-generic/arch/powerpc/odp_cpu_arch.c copy platform/linux-generic/{odp_barrier.c => drv_barrier.c} (59%) create mode 100644 platform/linux-generic/drv_shm.c create mode 100644 platform/linux-generic/include/_fdserver_internal.h create mode 100644 platform/linux-generic/include/_ishm_internal.h create mode 100644 platform/linux-generic/include/_ishmphy_internal.h create mode 100644 platform/linux-generic/include/ishmphy_internal.h create mode 100644 platform/linux-generic/include/odp/drv/barrier.h create mode 100644 platform/linux-generic/include/odp/drv/plat/barrier_types.h create mode 100644 platform/linux-generic/include/odp/drv/plat/shm_types.h create mode 100644 platform/linux-generic/include/odp/drv/plat/strong_types.h copy platform/linux-generic/include/odp/drv/{compiler.h => shm.h} (56%) create mode 100644 test/common_plat/validation/drv/drvshmem/.gitignore create mode 100644 test/common_plat/validation/drv/drvshmem/Makefile.am create mode 100644 test/common_plat/validation/drv/drvshmem/drvshmem.c create mode 100644 test/common_plat/validation/drv/drvshmem/drvshmem.h copy test/common_plat/validation/drv/{drvatomic/drvatomic_main.c => drvshmem/drvshmem_main.c} (72%) hooks/post-receive --

10 years

[COMMIT] branch api-next updated. v1.10.1.0-162-g0293146

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, api-next has been updated discards 9d4c9de84e2877cc5b03769f7bf5b1f3f2c93ec4 (commit) via 0293146428ef9c6d8d2b24e0bdd6c781486904a5 (commit) This update added new revisions after undoing existing revisions. That is to say, the old revision is not a strict subset of the new revision. This situation occurs when you --force push a change and generate a repository containing something like this: * -- * -- B -- O -- O -- O (9d4c9de84e2877cc5b03769f7bf5b1f3f2c93ec4) \ N -- N -- N (0293146428ef9c6d8d2b24e0bdd6c781486904a5) When this happens we assume that you've already had alert emails for all of the O revisions, and so we here report only the revisions in the N branch from the common base, B. Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 0293146428ef9c6d8d2b24e0bdd6c781486904a5 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Fri Aug 19 17:10:12 2016 +0200 doc: application-api-guide: excluding drv interface The input files parsed to generate the ODP platform specific API doc included (wrongly) part of the drv interface specification. This led to warning on undefined symbols. Fixed by restricting imput files to API only. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/doc/platform-api-guide/Doxyfile b/doc/platform-api-guide/Doxyfile index 93f6edf..fbe7c93 100644 --- a/doc/platform-api-guide/Doxyfile +++ b/doc/platform-api-guide/Doxyfile @@ -5,9 +5,9 @@ PROJECT_NUMBER = $(VERSION) PROJECT_LOGO = $(SRCDIR)/doc/images/ODP-Logo-HQ.svg INPUT = $(SRCDIR)/doc/application-api-guide \ $(SRCDIR)/doc/platform-api-guide \ - $(SRCDIR)/include \ + $(SRCDIR)/include/odp/api \ $(SRCDIR)/platform/$(WITH_PLATFORM)/doc \ - $(SRCDIR)/platform/$(WITH_PLATFORM)/include/odp \ + $(SRCDIR)/platform/$(WITH_PLATFORM)/include/odp/api \ $(SRCDIR)/platform/$(WITH_PLATFORM)/arch/$(WITH_ARCH) EXAMPLE_PATH = $(SRCDIR)/example $(SRCDIR)/platform $(SRCDIR) PREDEFINED = __GNUC__ \ ----------------------------------------------------------------------- Summary of changes: hooks/post-receive --

10 years

[COMMIT] branch api-next updated. v1.10.1.0-162-g9d4c9de

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, api-next has been updated via 9d4c9de84e2877cc5b03769f7bf5b1f3f2c93ec4 (commit) from a6d9960b81c1c5d4dff0c25b43ac5f9de7e36b8a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 9d4c9de84e2877cc5b03769f7bf5b1f3f2c93ec4 Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Fri Aug 19 17:10:12 2016 +0200 doc: application-api-guide: excluding drv interface The input files parsed to generate the ODP platform specific API doc included (wrongly) part of the drv interface specification. This led to warning on undefined symbols. Fixed by restricting imput files to API only. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/doc/platform-api-guide/Doxyfile b/doc/platform-api-guide/Doxyfile index 93f6edf..fbe7c93 100644 --- a/doc/platform-api-guide/Doxyfile +++ b/doc/platform-api-guide/Doxyfile @@ -5,9 +5,9 @@ PROJECT_NUMBER = $(VERSION) PROJECT_LOGO = $(SRCDIR)/doc/images/ODP-Logo-HQ.svg INPUT = $(SRCDIR)/doc/application-api-guide \ $(SRCDIR)/doc/platform-api-guide \ - $(SRCDIR)/include \ + $(SRCDIR)/include/odp/api \ $(SRCDIR)/platform/$(WITH_PLATFORM)/doc \ - $(SRCDIR)/platform/$(WITH_PLATFORM)/include/odp \ + $(SRCDIR)/platform/$(WITH_PLATFORM)/include/odp/api \ $(SRCDIR)/platform/$(WITH_PLATFORM)/arch/$(WITH_ARCH) EXAMPLE_PATH = $(SRCDIR)/example $(SRCDIR)/platform $(SRCDIR) PREDEFINED = __GNUC__ \ ----------------------------------------------------------------------- Summary of changes: doc/platform-api-guide/Doxyfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) hooks/post-receive --

10 years

[COMMIT] branch api-next updated. v1.10.1.0-161-ga6d9960

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, api-next has been updated via a6d9960b81c1c5d4dff0c25b43ac5f9de7e36b8a (commit) from 645e5ccf90c00c634c29fbb4276a8875699d8893 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit a6d9960b81c1c5d4dff0c25b43ac5f9de7e36b8a Author: Christophe Milard <christophe.milard(a)linaro.org> Date: Fri Aug 19 16:46:23 2016 +0200 doc: drv: fix doxyfile for drv doc generation Restricting to the driver interface and defining general macros. Signed-off-by: Christophe Milard <christophe.milard(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/doc/driver-api-guide/Doxyfile b/doc/driver-api-guide/Doxyfile index 680d1d4..b5f7260 100644 --- a/doc/driver-api-guide/Doxyfile +++ b/doc/driver-api-guide/Doxyfile @@ -5,10 +5,13 @@ PROJECT_NUMBER = $(VERSION) PROJECT_LOGO = $(SRCDIR)/doc/images/ODP-Logo-HQ.svg INPUT = $(SRCDIR)/doc/driver-api-guide \ $(SRCDIR)/include/odp/drv \ + $(SRCDIR)/platform/$(WITH_PLATFORM)/include/odp/drv \ $(SRCDIR)/include/odp_drv.h -EXCLUDE_PATTERNS = drv* odp_drv.h -EXAMPLE_PATH = $(SRCDIR)/example $(SRCDIR) PREDEFINED = __GNUC__ \ - "ODP_HANDLE_T(type)=odp_handle_t type" \ - odpdrv_bool_t=int -WARNINGS = NO + __attribute__(x)= \ + ODPDRV_ALIGNED(x)= \ + __BIG_ENDIAN_BITFIELD \ + __LITTLE_ENDIAN_BITFIELD \ + __x86_64__ \ + ODPDRV_PACKED \ + "ODPDRV_HANDLE_T(type)=odpdrv_handle_t type" ----------------------------------------------------------------------- Summary of changes: doc/driver-api-guide/Doxyfile | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) hooks/post-receive --

10 years

[COMMIT] branch master updated. v1.10.1.0-103-g7cd13f4

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 7cd13f48c935a88d56fd7d393fea6f02e5b0365a (commit) via ba2298af4f3912598043938c19bfedc5a8c89567 (commit) via 84c399d72a287115189bc8e8f272e4d7b9eaec47 (commit) from be6d2d4f3de94cc314d2367f76432306f4c88b94 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 7cd13f48c935a88d56fd7d393fea6f02e5b0365a Author: Xuelin Shi <forrest.shi(a)linaro.org> Date: Wed Aug 31 09:54:15 2016 +0800 example/l3fwd: enhance flow lookup performance in hash increase the hash size to accommodate more entries. pre-compute the hash keys of possible ip destinations in a subnet. Signed-off-by: Xuelin Shi <forrest.shi(a)linaro.org> Reviewed-and-tested-by: Matias Elo <matias.elo(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c index 0998614..44778b0 100644 --- a/example/l3fwd/odp_l3fwd.c +++ b/example/l3fwd/odp_l3fwd.c @@ -1061,9 +1061,6 @@ int main(int argc, char **argv) for (i = 0; i < nb_worker; i++) odph_odpthreads_join(&thread_tbl[i]); - /* release resource on exit */ - destroy_fwd_db(); - /* if_names share a single buffer, so only one free */ free(args->if_names[0]); diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c index b3a237f..4731237 100644 --- a/example/l3fwd/odp_l3fwd_db.c +++ b/example/l3fwd/odp_l3fwd_db.c @@ -150,7 +150,7 @@ char *mac_addr_str(char *b, odph_ethaddr_t *mac) */ typedef struct flow_entry_s { ipv4_tuple5_t key; /**< match key */ - struct flow_entry_s *next; /**< next entry on the list */ + struct flow_entry_s *next; /**< next entry in the bucket */ fwd_db_entry_t *fwd_entry; /**< entry info in db */ } flow_entry_t; @@ -158,59 +158,96 @@ typedef struct flow_entry_s { * Flow cache table bucket */ typedef struct flow_bucket_s { - odp_spinlock_t lock; /**< Bucket lock*/ - flow_entry_t *next; /**< Pointer to first flow entry in bucket*/ + odp_rwlock_t lock; /**< Bucket lock*/ + flow_entry_t *next; /**< First flow entry in bucket*/ } flow_bucket_t; /** * Flow hash table, fast lookup cache */ typedef struct flow_table_s { - flow_bucket_t *bucket; - uint32_t count; + odp_rwlock_t flow_lock; /**< flow table lock*/ + flow_entry_t *flows; /**< flow store */ + flow_bucket_t *bucket; /**< bucket store */ + uint32_t bkt_cnt; + uint32_t flow_cnt; + uint32_t next_flow; /**< next available flow in the store */ } flow_table_t; static flow_table_t fwd_lookup_cache; -void init_fwd_hash_cache(void) +static void create_fwd_hash_cache(void) { odp_shm_t hash_shm; flow_bucket_t *bucket; - uint32_t bucket_count; + flow_entry_t *flows; + uint32_t bucket_count, flow_count, size; uint32_t i; - bucket_count = FWD_DEF_BUCKET_COUNT; + flow_count = FWD_MAX_FLOW_COUNT; + bucket_count = flow_count / FWD_DEF_BUCKET_ENTRIES; /*Reserve memory for Routing hash table*/ - hash_shm = odp_shm_reserve("route_table", - sizeof(flow_bucket_t) * bucket_count, - ODP_CACHE_LINE_SIZE, 0); + size = sizeof(flow_bucket_t) * bucket_count + + sizeof(flow_entry_t) * flow_count; + hash_shm = odp_shm_reserve("flow_table", size, ODP_CACHE_LINE_SIZE, 0); bucket = odp_shm_addr(hash_shm); if (!bucket) { - EXAMPLE_ERR("Error: shared mem alloc failed.\n"); - exit(-1); + /* Try the second time with small request */ + flow_count /= 4; + bucket_count = flow_count / FWD_DEF_BUCKET_ENTRIES; + size = sizeof(flow_bucket_t) * bucket_count + + sizeof(flow_entry_t) * flow_count; + hash_shm = odp_shm_reserve("flow_table", size, + ODP_CACHE_LINE_SIZE, 0); + bucket = odp_shm_addr(hash_shm); + if (!bucket) { + EXAMPLE_ERR("Error: shared mem alloc failed.\n"); + exit(-1); + } } + size = sizeof(flow_bucket_t) * bucket_count; + flows = (flow_entry_t *)((char *)bucket + size); + fwd_lookup_cache.bucket = bucket; - fwd_lookup_cache.count = bucket_count; + fwd_lookup_cache.bkt_cnt = bucket_count; + fwd_lookup_cache.flows = flows; + fwd_lookup_cache.flow_cnt = flow_count; - /*Initialize Locks*/ + /*Initialize bucket locks*/ for (i = 0; i < bucket_count; i++) { bucket = &fwd_lookup_cache.bucket[i]; - odp_spinlock_init(&bucket->lock); + odp_rwlock_init(&bucket->lock); bucket->next = NULL; } + + memset(flows, 0, sizeof(flow_entry_t) * flow_count); + odp_rwlock_init(&fwd_lookup_cache.flow_lock); + fwd_lookup_cache.next_flow = 0; +} + +static inline flow_entry_t *get_new_flow(void) +{ + uint32_t next; + flow_entry_t *flow = NULL; + + odp_rwlock_write_lock(&fwd_lookup_cache.flow_lock); + next = fwd_lookup_cache.next_flow; + if (next < fwd_lookup_cache.flow_cnt) { + flow = &fwd_lookup_cache.flows[next]; + fwd_lookup_cache.next_flow++; + } + odp_rwlock_write_unlock(&fwd_lookup_cache.flow_lock); + + return flow; } static inline int match_key_flow(ipv4_tuple5_t *key, flow_entry_t *flow) { - if (key->src_ip == flow->key.src_ip && - key->dst_ip == flow->key.dst_ip && - key->src_port == flow->key.src_port && - key->dst_port == flow->key.dst_port && - key->proto == flow->key.proto) + if (key->hi64 == flow->key.hi64 && key->lo64 == flow->key.lo64) return 1; return 0; @@ -221,10 +258,12 @@ flow_entry_t *lookup_fwd_cache(ipv4_tuple5_t *key, flow_bucket_t *bucket) { flow_entry_t *rst; + odp_rwlock_read_lock(&bucket->lock); for (rst = bucket->next; rst != NULL; rst = rst->next) { if (match_key_flow(key, rst)) break; } + odp_rwlock_read_unlock(&bucket->lock); return rst; } @@ -239,22 +278,58 @@ flow_entry_t *insert_fwd_cache(ipv4_tuple5_t *key, if (!entry) return NULL; - flow = malloc(sizeof(flow_entry_t)); + flow = get_new_flow(); + if (!flow) + return NULL; + flow->key = *key; flow->fwd_entry = entry; - odp_spinlock_lock(&bucket->lock); - if (!bucket->next) { - bucket->next = flow; - } else { + odp_rwlock_write_lock(&bucket->lock); + if (bucket->next) flow->next = bucket->next; - bucket->next = flow; - } - odp_spinlock_unlock(&bucket->lock); + bucket->next = flow; + odp_rwlock_write_unlock(&bucket->lock); return flow; } +void init_fwd_hash_cache(void) +{ + fwd_db_entry_t *entry; + flow_entry_t *flow; + flow_bucket_t *bucket; + uint64_t hash; + uint32_t i, nb_hosts; + ipv4_tuple5_t key; + + create_fwd_hash_cache(); + + /** + * warm up the lookup cache with possible hosts. + * with millions flows, save significant time during runtime. + */ + memset(&key, 0, sizeof(key)); + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { + nb_hosts = 1 << (32 - entry->subnet.depth); + for (i = 0; i < nb_hosts; i++) { + key.dst_ip = entry->subnet.addr + i; + hash = l3fwd_calc_hash(&key); + hash &= fwd_lookup_cache.bkt_cnt - 1; + bucket = &fwd_lookup_cache.bucket[hash]; + flow = lookup_fwd_cache(&key, bucket); + if (flow) + return; + + flow = insert_fwd_cache(&key, bucket, entry); + if (!flow) + goto out; + } + } +out: + return; +} + /** Global pointer to fwd db */ fwd_db_t *fwd_db; @@ -382,35 +457,22 @@ void dump_fwd_db(void) printf("\n"); } -void destroy_fwd_db(void) -{ - flow_bucket_t *bucket; - flow_entry_t *flow, *tmp; - uint32_t i; - - for (i = 0; i < fwd_lookup_cache.count; i++) { - bucket = &fwd_lookup_cache.bucket[i]; - flow = bucket->next; - odp_spinlock_lock(&bucket->lock); - while (flow) { - tmp = flow->next; - free(flow); - flow = tmp; - } - odp_spinlock_unlock(&bucket->lock); - } -} - fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key) { fwd_db_entry_t *entry; flow_entry_t *flow; flow_bucket_t *bucket; uint64_t hash; + ipv4_tuple5_t newkey; + + newkey.hi64 = 0; + newkey.lo64 = 0; + newkey.dst_ip = key->dst_ip; + key = &newkey; /* first find in cache */ hash = l3fwd_calc_hash(key); - hash &= fwd_lookup_cache.count - 1; + hash &= fwd_lookup_cache.bkt_cnt - 1; bucket = &fwd_lookup_cache.bucket[hash]; flow = lookup_fwd_cache(key, bucket); if (flow) diff --git a/example/l3fwd/odp_l3fwd_db.h b/example/l3fwd/odp_l3fwd_db.h index d9f6d61..1b24f1a 100644 --- a/example/l3fwd/odp_l3fwd_db.h +++ b/example/l3fwd/odp_l3fwd_db.h @@ -19,14 +19,14 @@ extern "C" { #define MAX_STRING 32 /** - * Default number of flows + * Max number of flows */ -#define FWD_DEF_FLOW_COUNT 100000 +#define FWD_MAX_FLOW_COUNT (1 << 22) /** - * Default Hash bucket number + * Default hash entries in a bucket */ -#define FWD_DEF_BUCKET_COUNT (FWD_DEF_FLOW_COUNT / 8) +#define FWD_DEF_BUCKET_ENTRIES 4 /** * IP address range (subnet) @@ -40,12 +40,22 @@ typedef struct ip_addr_range_s { * TCP/UDP flow */ typedef struct ipv4_tuple5_s { - uint32_t src_ip; - uint32_t dst_ip; - uint16_t src_port; - uint16_t dst_port; - uint8_t proto; -} ipv4_tuple5_t; + union { + struct { + int32_t src_ip; + int32_t dst_ip; + int16_t src_port; + int16_t dst_port; + int8_t proto; + int8_t pad1; + int16_t pad2; + }; + struct { + int64_t hi64; + int64_t lo64; + }; + }; +} ipv4_tuple5_t ODP_ALIGNED_CACHE; /** * Forwarding data base entry @@ -116,11 +126,6 @@ void dump_fwd_db_entry(fwd_db_entry_t *entry); void dump_fwd_db(void); /** - * Destroy the forwarding database - */ -void destroy_fwd_db(void); - -/** * Find a matching forwarding database entry * * @param key ipv4 tuple commit ba2298af4f3912598043938c19bfedc5a8c89567 Author: Xuelin Shi <forrest.shi(a)linaro.org> Date: Wed Aug 31 09:54:14 2016 +0800 example/l3fwd: add test script Signed-off-by: Xuelin Shi <forrest.shi(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am index f926717..422a3bf 100644 --- a/example/l3fwd/Makefile.am +++ b/example/l3fwd/Makefile.am @@ -10,3 +10,11 @@ noinst_HEADERS = \ $(top_srcdir)/example/example_debug.h dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c odp_l3fwd_lpm.c + +if test_example +if HAVE_PCAP +TESTS = odp_l3fwd_run.sh +endif +endif + +EXTRA_DIST = odp_l3fwd_run.sh udp64.pcap diff --git a/example/l3fwd/odp_l3fwd_run.sh b/example/l3fwd/odp_l3fwd_run.sh new file mode 100755 index 0000000..6f0b11a --- /dev/null +++ b/example/l3fwd/odp_l3fwd_run.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# +# Copyright (c) 2016, Linaro Limited +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +# + +PCAP_IN=`find . ${TEST_DIR} $(dirname $0) -name udp64.pcap -print -quit` +PCAP_OUT="pcapout.pcap" +PCAP_IN_SIZE=`stat -c %s ${PCAP_IN}` +echo "using PCAP_IN = ${PCAP_IN}, PCAP_OUT = ${PCAP_OUT}" + +./odp_l3fwd -i pcap:in=${PCAP_IN},pcap:out=${PCAP_OUT} \ + -r "10.0.0.0/24,pcap:out=${PCAP_OUT}" -d 30 + +STATUS=$? +PCAP_OUT_SIZE=`stat -c %s ${PCAP_OUT}` +rm -f ${PCAP_OUT} + +if [ ${STATUS} -ne 0 ] || [ ${PCAP_IN_SIZE} -ne ${PCAP_OUT_SIZE} ]; then + echo "Error: status ${STATUS}, in:${PCAP_IN_SIZE} out:${PCAP_OUT_SIZE}" + exit 1 +fi + +echo "Pass: status ${STATUS}, in:${PCAP_IN_SIZE} out:${PCAP_OUT_SIZE}" + +exit 0 diff --git a/example/l3fwd/udp64.pcap b/example/l3fwd/udp64.pcap new file mode 100644 index 0000000..45f9d6e Binary files /dev/null and b/example/l3fwd/udp64.pcap differ commit 84c399d72a287115189bc8e8f272e4d7b9eaec47 Author: Xuelin Shi <forrest.shi(a)linaro.org> Date: Wed Aug 31 09:54:13 2016 +0800 example: introducing l3fwd multi-thread, multi-queues and bi-directional forwarding. support (port, queue, thread) arguments in cmdline which specify how the threads handle which rx queue at which port, if no this argument, default specification used. both hash and lpm based lookup methods are supported, default lpm. Signed-off-by: Xuelin Shi <forrest.shi(a)linaro.org> Reviewed-and-tested-by: Matias Elo <matias.elo(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/example/Makefile.am b/example/Makefile.am index 37542af..dfc07b6 100644 --- a/example/Makefile.am +++ b/example/Makefile.am @@ -1 +1,11 @@ -SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt l2fwd_simple switch hello +SUBDIRS = classifier \ + generator \ + hello \ + ipsec \ + l2fwd_simple \ + l3fwd \ + packet \ + switch \ + time \ + timer \ + traffic_mgmt diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.gitignore new file mode 100644 index 0000000..74e501f --- /dev/null +++ b/example/l3fwd/.gitignore @@ -0,0 +1 @@ +odp_l3fwd diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am new file mode 100644 index 0000000..f926717 --- /dev/null +++ b/example/l3fwd/Makefile.am @@ -0,0 +1,12 @@ +include $(top_srcdir)/example/Makefile.inc + +bin_PROGRAMS = odp_l3fwd$(EXEEXT) +odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static +odp_l3fwd_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example -I${top_srcdir}/test + +noinst_HEADERS = \ + $(top_srcdir)/example/l3fwd/odp_l3fwd_db.h \ + $(top_srcdir)/example/l3fwd/odp_l3fwd_lpm.h \ + $(top_srcdir)/example/example_debug.h + +dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c odp_l3fwd_lpm.c diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c new file mode 100644 index 0000000..0998614 --- /dev/null +++ b/example/l3fwd/odp_l3fwd.c @@ -0,0 +1,1089 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <stdlib.h> +#include <stdio.h> +#include <errno.h> +#include <getopt.h> +#include <unistd.h> +#include <inttypes.h> + +#include <test_debug.h> + +#include <odp_api.h> +#include <odp/helper/linux.h> +#include <odp/helper/eth.h> +#include <odp/helper/ip.h> +#include <odp/helper/udp.h> +#include <odp/helper/tcp.h> + +#include "odp_l3fwd_db.h" +#include "odp_l3fwd_lpm.h" + +#define POOL_NUM_PKT 8192 +#define POOL_SEG_LEN 1856 +#define MAX_PKT_BURST 32 + +#define MAX_NB_WORKER 32 +#define MAX_NB_PKTIO 32 +#define MAX_NB_QUEUE 32 +#define MAX_NB_QCONFS 1024 +#define MAX_NB_ROUTE 32 + +#define INVALID_ID (-1) +#define PRINT_INTERVAL 10 /* interval seconds of printing stats */ + +/** Get rid of path in filename - only for unix-type paths using '/' */ +#define NO_PATH(file_name) (strrchr((file_name), '/') ? \ + strrchr((file_name), '/') + 1 : (file_name)) + +struct l3fwd_pktio_s { + odp_pktio_t pktio; + odph_ethaddr_t mac_addr; + odp_pktin_queue_t ifin[MAX_NB_QUEUE]; + odp_pktout_queue_t ifout[MAX_NB_QUEUE]; + int nb_rxq; /* capa max */ + int nb_txq; /* capa max */ + int rxq_idx; /* requested, maybe greater than nb_rxq */ + int txq_idx; /* requested, maybe greater than nb_txq */ +}; + +struct l3fwd_qconf_s { + uint8_t if_idx; /* port index */ + uint8_t rxq_idx; /* recv queue index in a port */ + uint8_t core_idx; /* this core should handle traffic */ +}; + +struct thread_arg_s { + uint64_t packets; + uint64_t rx_drops; + uint64_t tx_drops; + struct { + int nb_rxq; /* number of rxq this thread will access */ + int rxq[MAX_NB_QUEUE]; /* rxq[i] is index in pktio.ifin[] */ + int txq_idx; /* index in pktio.ifout[] */ + } pktio[MAX_NB_PKTIO]; + int nb_pktio; + int thr_idx; +}; + +typedef struct { + char *if_names[MAX_NB_PKTIO]; + int if_count; + char *route_str[MAX_NB_ROUTE]; + int worker_count; + struct l3fwd_qconf_s qconf_config[MAX_NB_QCONFS]; + int qconf_count; + uint32_t duration; /* seconds to run */ + uint8_t hash_mode; /* 1:hash, 0:lpm */ + uint8_t dest_mac_changed[MAX_NB_PKTIO]; /* 1: dest mac from cmdline */ +} app_args_t; + +struct { + app_args_t cmd_args; + struct l3fwd_pktio_s l3fwd_pktios[MAX_NB_PKTIO]; + odph_odpthread_t l3fwd_workers[MAX_NB_WORKER]; + struct thread_arg_s worker_args[MAX_NB_WORKER]; + odph_ethaddr_t eth_dest_mac[MAX_NB_PKTIO]; + + /* forward func, hash or lpm */ + int (*fwd_func)(odp_packet_t pkt, int sif); +} global; + +/** Global barrier to synchronize main and workers */ +static odp_barrier_t barrier; +static int exit_threads; /**< Break workers loop if set to 1 */ + +static int create_pktio(const char *name, odp_pool_t pool, + struct l3fwd_pktio_s *fwd_pktio) +{ + odp_pktio_param_t pktio_param; + odp_pktio_t pktio; + odp_pktio_capability_t capa; + int rc; + + odp_pktio_param_init(&pktio_param); + + pktio = odp_pktio_open(name, pool, &pktio_param); + if (pktio == ODP_PKTIO_INVALID) { + printf("Failed to open %s\n", name); + return -1; + } + fwd_pktio->pktio = pktio; + + rc = odp_pktio_capability(pktio, &capa); + if (rc) { + printf("Error: pktio %s: unable to read capabilities!\n", + name); + + return -1; + } + + fwd_pktio->nb_rxq = (int)capa.max_input_queues; + fwd_pktio->nb_txq = (int)capa.max_output_queues; + + if (fwd_pktio->nb_rxq > MAX_NB_QUEUE) + fwd_pktio->nb_rxq = MAX_NB_QUEUE; + + if (fwd_pktio->nb_txq > MAX_NB_QUEUE) + fwd_pktio->nb_txq = MAX_NB_QUEUE; + + return 0; +} + +static void setup_fwd_db(void) +{ + fwd_db_entry_t *entry; + int if_idx; + app_args_t *args; + + args = &global.cmd_args; + if (args->hash_mode) + init_fwd_hash_cache(); + else + fib_tbl_init(); + + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { + if_idx = entry->oif_id; + if (!args->hash_mode) + fib_tbl_insert(entry->subnet.addr, if_idx, + entry->subnet.depth); + if (args->dest_mac_changed[if_idx]) + global.eth_dest_mac[if_idx] = entry->dst_mac; + else + entry->dst_mac = global.eth_dest_mac[if_idx]; + } +} + +static int l3fwd_pkt_hash(odp_packet_t pkt, int sif) +{ + fwd_db_entry_t *entry; + ipv4_tuple5_t key; + odph_ethhdr_t *eth; + odph_udphdr_t *udp; + odph_ipv4hdr_t *ip; + uint32_t len; + int dif; + + ip = odp_packet_l3_ptr(pkt, &len); + key.dst_ip = odp_be_to_cpu_32(ip->dst_addr); + key.src_ip = odp_be_to_cpu_32(ip->src_addr); + key.proto = ip->proto; + + if (odp_packet_has_udp(pkt) || + odp_packet_has_tcp(pkt)) { + /* UDP or TCP*/ + void *ptr = odp_packet_l4_ptr(pkt, NULL); + + udp = (odph_udphdr_t *)ptr; + key.src_port = odp_be_to_cpu_16(udp->src_port); + key.dst_port = odp_be_to_cpu_16(udp->dst_port); + } else { + key.src_port = 0; + key.dst_port = 0; + } + entry = find_fwd_db_entry(&key); + ip->ttl--; + ip->chksum = odph_ipv4_csum_update(pkt); + eth = odp_packet_l2_ptr(pkt, NULL); + if (entry) { + eth->src = entry->src_mac; + eth->dst = entry->dst_mac; + dif = entry->oif_id; + } else { + /* no route, send by src port */ + eth->dst = eth->src; + dif = sif; + } + + return dif; +} + +static int l3fwd_pkt_lpm(odp_packet_t pkt, int sif) +{ + odph_ipv4hdr_t *ip; + odph_ethhdr_t *eth; + uint32_t len; + int dif; + int ret; + + ip = odp_packet_l3_ptr(pkt, &len); + ip->ttl--; + ip->chksum = odph_ipv4_csum_update(pkt); + eth = odp_packet_l2_ptr(pkt, NULL); + + /* network byte order maybe different from host */ + ret = fib_tbl_lookup(odp_be_to_cpu_32(ip->dst_addr), &dif); + if (ret) + dif = sif; + + eth->dst = global.eth_dest_mac[dif]; + eth->src = global.l3fwd_pktios[dif].mac_addr; + + return dif; +} + +/** + * Drop packets which input parsing marked as containing errors. + * + * Frees packets with error and modifies pkt_tbl[] to only contain packets with + * no detected errors. + * + * @param pkt_tbl Array of packets + * @param num Number of packets in pkt_tbl[] + * + * @return Number of packets dropped + */ +static inline int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned num) +{ + odp_packet_t pkt; + unsigned dropped = 0; + unsigned i, j; + + for (i = 0, j = 0; i < num; ++i) { + pkt = pkt_tbl[i]; + + if (odp_unlikely(odp_packet_has_error(pkt) || + !odp_packet_has_ipv4(pkt))) { + odp_packet_free(pkt); + dropped++; + } else if (odp_unlikely(i != j++)) { + pkt_tbl[j - 1] = pkt; + } + } + + return dropped; +} + +static void l3fwd_one_queue(uint32_t sif, int rxq_idx, void *thr_arg) +{ + odp_packet_t *tbl; + odp_pktout_queue_t outq; + odp_pktin_queue_t inq; + odp_packet_t pkt_tbl[MAX_PKT_BURST]; + struct thread_arg_s *arg; + int pkts, drop, sent; + int dst_port, dif; + int i; + + arg = thr_arg; + inq = global.l3fwd_pktios[sif].ifin[rxq_idx]; + pkts = odp_pktin_recv(inq, pkt_tbl, MAX_PKT_BURST); + if (pkts <= 0) + return; + + arg->packets += pkts; + drop = drop_err_pkts(pkt_tbl, pkts); + pkts -= drop; + arg->rx_drops += drop; + + dif = global.fwd_func(pkt_tbl[0], sif); + tbl = &pkt_tbl[0]; + while (pkts) { + int txq_idx; + + dst_port = dif; + for (i = 1; i < pkts; i++) { + dif = global.fwd_func(tbl[i], sif); + if (dif != dst_port) + break; + } + + txq_idx = arg->pktio[dst_port].txq_idx; + outq = global.l3fwd_pktios[dst_port].ifout[txq_idx]; + sent = odp_pktout_send(outq, tbl, i); + if (odp_unlikely(sent < i)) { + sent = sent < 0 ? 0 : sent; + odp_packet_free_multi(&tbl[sent], i - sent); + arg->tx_drops += i - sent; + } + + if (i < pkts) + tbl += i; + + pkts -= i; + } +} + +static int run_worker(void *arg) +{ + int if_idx, rxq, nb_rxq; + struct thread_arg_s *thr_arg = arg; + + odp_barrier_wait(&barrier); + + while (!exit_threads) { + for (if_idx = 0; if_idx < thr_arg->nb_pktio; if_idx++) { + nb_rxq = thr_arg->pktio[if_idx].nb_rxq; + if (!nb_rxq || thr_arg->thr_idx == INVALID_ID) + continue; + + for (rxq = 0; rxq < nb_rxq; rxq++) { + int rxq_idx; + + rxq_idx = thr_arg->pktio[if_idx].rxq[rxq]; + l3fwd_one_queue(if_idx, rxq_idx, arg); + } + } + } + + /* Make sure that latest stat writes are visible to other threads */ + odp_mb_full(); + + return 0; +} + +static int find_port_id_by_name(char *name, app_args_t *args) +{ + int i; + + if (!name) + return -1; + + for (i = 0; i < args->if_count; i++) { + if (!strcmp(name, args->if_names[i])) + return i; + } + + return -1; +} + +/* split string into tokens */ +static int split_string(char *str, int stringlen, + char **tokens, int maxtokens, char delim) +{ + int i, tok = 0; + int tokstart = 1; /* first token is right at start of string */ + + if (str == NULL || tokens == NULL) + goto einval_error; + + for (i = 0; i < stringlen; i++) { + if (str[i] == '\0' || tok >= maxtokens) + break; + if (tokstart) { + tokstart = 0; + tokens[tok++] = &str[i]; + } + if (str[i] == delim) { + str[i] = '\0'; + tokstart = 1; + } + } + return tok; + +einval_error: + errno = EINVAL; + return -1; +} + +static int parse_config(char *cfg_str, app_args_t *args) +{ + char s[256]; + const char *p, *p0 = cfg_str; + char *end; + enum fieldnames { + FLD_PORT = 0, + FLD_QUEUE, + FLD_LCORE, + FLD_LAST + }; + unsigned long int_fld[FLD_LAST]; + char *str_fld[FLD_LAST]; + int i; + unsigned size; + int nb_qconfs = 0; + struct l3fwd_qconf_s *qconf_array = &args->qconf_config[0]; + + p = strchr(p0, '('); + while (p != NULL) { + ++p; + p0 = strchr(p, ')'); + if (p0 == NULL) + return -1; + + size = p0 - p; + if (size >= sizeof(s)) + return -1; + + snprintf(s, sizeof(s), "%.*s", size, p); + i = split_string(s, sizeof(s), str_fld, FLD_LAST, ','); + if (i != FLD_LAST) + return -1; + for (i = 0; i < FLD_LAST; i++) { + errno = 0; + int_fld[i] = strtoul(str_fld[i], &end, 0); + if (errno != 0 || end == str_fld[i] || int_fld[i] > 255) + return -1; + } + if (nb_qconfs >= MAX_NB_QCONFS) { + printf("exceeded max number of queue params: %d\n", + nb_qconfs); + return -1; + } + qconf_array[nb_qconfs].if_idx = (uint8_t)int_fld[FLD_PORT]; + qconf_array[nb_qconfs].rxq_idx = (uint8_t)int_fld[FLD_QUEUE]; + qconf_array[nb_qconfs].core_idx = (uint8_t)int_fld[FLD_LCORE]; + ++nb_qconfs; + + p = strchr(p0, '('); + } + args->qconf_count = nb_qconfs; + + return 0; +} + +static void print_usage(char *progname) +{ + printf("\n" + "ODP L3 forwarding application.\n" + "\n" + "Usage: %s OPTIONS\n" + " E.g. %s -i eth0,eth1 -r 1.1.1.0/24,eth0 -r 2.2.2.0/24,eth1\n" + " In the above example,\n" + " eth0 will send pkts to eth1 and vice versa\n" + "\n" + "Mandatory OPTIONS:\n" + " -i, --interface eth interfaces (comma-separated, no spaces)\n" + " -r, --route SubNet,Intf[,NextHopMAC]\n" + " NextHopMAC can be optional\n" + "\n" + "Optional OPTIONS:\n" + " -s, --style [lpm|hash], ip lookup method\n" + " optional, default as lpm\n" + " -d, --duration Seconds to run and print stats\n" + " optional, default as 0, run forever\n" + " -t, --thread Number of threads to do forwarding\n" + " optional, default as availbe worker cpu count\n" + " -q, --queue Configure rx queue(s) for port\n" + " optional, format: [(port, queue, thread),...]\n" + " for example: -q '(0, 0, 1),(1,0,2)'\n" + " -h, --help Display help and exit.\n\n" + "\n", NO_PATH(progname), NO_PATH(progname) + ); +} + +static void parse_cmdline_args(int argc, char *argv[], app_args_t *args) +{ + int opt; + int long_index; + char *token, *local; + size_t len, route_index = 0; + int i, mem_failure = 0; + + static struct option longopts[] = { + {"interface", required_argument, NULL, 'i'}, /* return 'i' */ + {"route", required_argument, NULL, 'r'}, /* return 'r' */ + {"style", optional_argument, NULL, 's'}, /* return 's' */ + {"duration", optional_argument, NULL, 'd'}, /* return 'd' */ + {"thread", optional_argument, NULL, 't'}, /* return 't' */ + {"queue", optional_argument, NULL, 'q'}, /* return 'q' */ + {"help", no_argument, NULL, 'h'}, /* return 'h' */ + {NULL, 0, NULL, 0} + }; + + while (1) { + opt = getopt_long(argc, argv, "+s:t:d:i:r:q:h", + longopts, &long_index); + + if (opt == -1) + break; /* No more options */ + + switch (opt) { + /* parse ip lookup method */ + case 's': + if (!strcmp(optarg, "hash")) + args->hash_mode = 1; + break; + /* parse number of worker threads to be run*/ + case 't': + i = odp_cpu_count(); + args->worker_count = atoi(optarg); + if (args->worker_count > i) { + printf("Too many threads," + "truncate to cpu count: %d\n", i); + args->worker_count = i; + } + + break; + + /* parse seconds to run */ + case 'd': + args->duration = atoi(optarg); + break; + + /* parse packet-io interface names */ + case 'i': + len = strlen(optarg); + if (len == 0) { + print_usage(argv[0]); + exit(EXIT_FAILURE); + } + len += 1; /* add room for '\0' */ + + local = malloc(len); + if (!local) { + print_usage(argv[0]); + exit(EXIT_FAILURE); + } + + /* count the number of tokens separated by ',' */ + strcpy(local, optarg); + for (token = strtok(local, ","), i = 0; + token != NULL; + token = strtok(NULL, ","), i++) + ; + + if (i == 0) { + print_usage(argv[0]); + free(local); + exit(EXIT_FAILURE); + } else if (i > MAX_NB_PKTIO) { + printf("too many ports specified, " + "truncated to %d", MAX_NB_PKTIO); + } + args->if_count = i; + + /* store the if names (reset names string) */ + strcpy(local, optarg); + for (token = strtok(local, ","), i = 0; + token != NULL; token = strtok(NULL, ","), i++) { + args->if_names[i] = token; + } + break; + + /*Configure Route in forwarding database*/ + case 'r': + if (route_index >= MAX_NB_ROUTE) { + printf("No more routes can be added\n"); + break; + } + local = calloc(1, strlen(optarg) + 1); + if (!local) { + mem_failure = 1; + break; + } + memcpy(local, optarg, strlen(optarg)); + local[strlen(optarg)] = '\0'; + args->route_str[route_index++] = local; + break; + + case 'h': + print_usage(argv[0]); + exit(EXIT_SUCCESS); + break; + + case 'q': + parse_config(optarg, args); + break; + + default: + break; + } + } + + /* checking arguments */ + if (args->if_count == 0) { + printf("\nNo option -i specified.\n"); + goto out; + } + + if (args->route_str[0] == NULL) { + printf("\nNo option -r specified.\n"); + goto out; + } + + if (mem_failure == 1) { + printf("\nAllocate memory failure.\n"); + goto out; + } + optind = 1; /* reset 'extern optind' from the getopt lib */ + return; + +out: + print_usage(argv[0]); + exit(EXIT_FAILURE); +} + +static void print_info(char *progname, app_args_t *args) +{ + int i; + + printf("\n" + "ODP system info\n" + "---------------\n" + "ODP API version: %s\n" + "ODP impl name: %s\n" + "CPU model: %s\n" + "CPU freq (hz): %" PRIu64 "\n" + "Cache line size: %i\n" + "CPU count: %i\n" + "\n", + odp_version_api_str(), odp_version_impl_name(), + odp_cpu_model_str(), odp_cpu_hz_max(), + odp_sys_cache_line_size(), odp_cpu_count()); + + printf("Running ODP appl: \"%s\"\n" + "-----------------\n" + "IP Lookup: %s\n" + "IF Count: %i\n" + "Using IFs: ", + progname, + args->hash_mode ? "hash" : "lpm", + args->if_count); + + for (i = 0; i < args->if_count; ++i) + printf(" %s", args->if_names[i]); + + printf("\n\n"); + fflush(NULL); +} + +/** + * Setup rx and tx queues, distribute them among threads. + * + * If no q argument, the queues are distribute among threads as default. + * The thread take one rx queue of a port one time as round-robin order. + * One txq for each thread on each port + */ +static void setup_worker_qconf(app_args_t *args) +{ + int nb_worker, if_count; + int i, j, rxq_idx; + struct thread_arg_s *arg; + struct l3fwd_pktio_s *port; + uint8_t queue_mask[MAX_NB_PKTIO][MAX_NB_QUEUE]; + + nb_worker = args->worker_count; + if_count = args->if_count; + + /* distribute rx queues among threads as round-robin */ + if (!args->qconf_count) { + if (nb_worker > if_count) { + for (i = 0; i < nb_worker; i++) { + arg = &global.worker_args[i]; + arg->thr_idx = i; + j = i % if_count; + port = &global.l3fwd_pktios[j]; + rxq_idx = arg->pktio[j].nb_rxq; + arg->pktio[j].rxq[rxq_idx] = + port->rxq_idx % port->nb_rxq; + arg->pktio[j].nb_rxq++; + port->rxq_idx++; + } + } else { + for (i = 0; i < if_count; i++) { + j = i % nb_worker; + arg = &global.worker_args[j]; + arg->thr_idx = j; + port = &global.l3fwd_pktios[i]; + rxq_idx = arg->pktio[i].nb_rxq; + arg->pktio[i].rxq[rxq_idx] = + port->rxq_idx % port->nb_rxq; + arg->pktio[i].nb_rxq++; + port->rxq_idx++; + } + } + } + + /* distribute rx queues among threads as q argument */ + memset(queue_mask, 0, sizeof(queue_mask)); + for (i = 0; i < args->qconf_count; i++) { + struct l3fwd_qconf_s *q; + + q = &args->qconf_config[i]; + if (q->core_idx >= nb_worker || q->if_idx >= if_count) + LOG_ABORT("Error queue (%d, %d, %d), max port: " + "%d, max core: %d\n", q->if_idx, q->rxq_idx, + q->core_idx, args->if_count - 1, + args->worker_count - 1); + + /* check if one queue is configured twice or more */ + if (queue_mask[q->if_idx][q->rxq_idx]) + LOG_ABORT("Error queue (%d, %d, %d), reconfig queue\n", + q->if_idx, q->rxq_idx, q->core_idx); + queue_mask[q->if_idx][q->rxq_idx] = 1; + + port = &global.l3fwd_pktios[q->if_idx]; + if (port->rxq_idx < q->rxq_idx) + LOG_ABORT("Error queue (%d, %d, %d), queue should be" + " in sequence and start from 0, queue %d\n", + q->if_idx, q->rxq_idx, q->core_idx, + q->rxq_idx); + + if (q->rxq_idx > port->nb_rxq) { + LOG_ABORT("Error queue (%d, %d, %d), max queue %d\n", + q->if_idx, q->rxq_idx, q->core_idx, + port->nb_rxq - 1); + } + port->rxq_idx = q->rxq_idx + 1; + + /* put the queue into worker_args */ + arg = &global.worker_args[q->core_idx]; + rxq_idx = arg->pktio[q->if_idx].nb_rxq; + arg->pktio[q->if_idx].rxq[rxq_idx] = q->rxq_idx; + arg->pktio[q->if_idx].nb_rxq++; + arg->thr_idx = q->core_idx; + } + + /* distribute tx queues among threads */ + for (i = 0; i < args->worker_count; i++) { + arg = &global.worker_args[i]; + for (j = 0; j < args->if_count; j++) { + port = &global.l3fwd_pktios[j]; + arg->pktio[j].txq_idx = + port->txq_idx % port->nb_txq; + port->txq_idx++; + } + } + + /* config and initialize rx and tx queues. */ + for (i = 0; i < if_count; i++) { + odp_pktin_queue_param_t in_queue_param; + odp_pktout_queue_param_t out_queue_param; + odp_pktin_queue_t *inq; + odp_pktout_queue_t *outq; + const char *name; + int nb_rxq, nb_txq; + + port = &global.l3fwd_pktios[i]; + name = args->if_names[i]; + odp_pktin_queue_param_init(&in_queue_param); + odp_pktout_queue_param_init(&out_queue_param); + + in_queue_param.op_mode = ODP_PKTIO_OP_MT_UNSAFE; + out_queue_param.op_mode = ODP_PKTIO_OP_MT_UNSAFE; + + in_queue_param.hash_enable = 1; + in_queue_param.hash_proto.proto.ipv4 = 1; + in_queue_param.hash_proto.proto.ipv4_tcp = 1; + in_queue_param.hash_proto.proto.ipv4_udp = 1; + + in_queue_param.num_queues = port->rxq_idx; + if (port->rxq_idx > port->nb_rxq) { + in_queue_param.num_queues = port->nb_rxq; + in_queue_param.op_mode = ODP_PKTIO_OP_MT; + } + + if (odp_pktin_queue_config(port->pktio, &in_queue_param)) + LOG_ABORT("Fail to config input queue for port %s\n", + name); + + out_queue_param.num_queues = port->txq_idx; + if (port->txq_idx > port->nb_txq) { + out_queue_param.num_queues = port->nb_txq; + out_queue_param.op_mode = ODP_PKTIO_OP_MT; + } + if (odp_pktout_queue_config(port->pktio, &out_queue_param)) + LOG_ABORT("Fail to config output queue for port %s\n", + name); + + inq = port->ifin; + nb_rxq = in_queue_param.num_queues; + if (odp_pktin_queue(port->pktio, inq, nb_rxq) != nb_rxq) + LOG_ABORT("Fail to set pktin queue for port %s\n", + name); + + outq = port->ifout; + nb_txq = out_queue_param.num_queues; + if (odp_pktout_queue(port->pktio, outq, nb_txq) != nb_txq) + LOG_ABORT("Fail to set pktout queue for port %s\n", + name); + } +} + +static void print_qconf_table(app_args_t *args) +{ + int i, j, k, qid; + char buf[32]; + struct thread_arg_s *thr_arg; + + printf("Rx Queue table\n" + "-----------------\n" + "%-32s%-16s%-16s\n", + "port/id", "rxq", "thread"); + + for (i = 0; i < args->worker_count; i++) { + thr_arg = &global.worker_args[i]; + for (j = 0; j < args->if_count; j++) { + if (!thr_arg->pktio[j].nb_rxq) + continue; + + snprintf(buf, 32, "%s/%d", args->if_names[j], j); + for (k = 0; k < MAX_NB_QUEUE; k++) { + qid = thr_arg->pktio[j].rxq[k]; + if (qid != INVALID_ID) + printf("%-32s%-16d%-16d\n", buf, qid, + thr_arg->thr_idx); + } + } + } + printf("\n"); + fflush(NULL); +} + +/** + * Print statistics + * + * @param num_workers Number of worker threads + * @param duration Number of seconds to loop in + * @param timeout Number of seconds for stats calculation + * + */ +static int print_speed_stats(int num_workers, int duration, int timeout) +{ + uint64_t pkts = 0; + uint64_t pkts_prev = 0; + uint64_t pps; + uint64_t rx_drops, tx_drops; + uint64_t maximum_pps = 0; + int i; + int elapsed = 0; + int stats_enabled = 1; + int loop_forever = (duration == 0); + + if (timeout <= 0) { + stats_enabled = 0; + timeout = 1; + } + /* Wait for all threads to be ready*/ + odp_barrier_wait(&barrier); + + do { + pkts = 0; + rx_drops = 0; + tx_drops = 0; + sleep(timeout); + + for (i = 0; i < num_workers; i++) { + pkts += global.worker_args[i].packets; + rx_drops += global.worker_args[i].rx_drops; + tx_drops += global.worker_args[i].tx_drops; + } + if (stats_enabled) { + pps = (pkts - pkts_prev) / timeout; + if (pps > maximum_pps) + maximum_pps = pps; + printf("%" PRIu64 " pps, %" PRIu64 " max pps, ", pps, + maximum_pps); + + printf(" %" PRIu64 " rx drops, %" PRIu64 " tx drops\n", + rx_drops, tx_drops); + + pkts_prev = pkts; + } + elapsed += timeout; + } while (loop_forever || (elapsed < duration)); + + if (stats_enabled) + printf("TEST RESULT: %" PRIu64 " maximum packets per second.\n", + maximum_pps); + + return pkts > 100 ? 0 : -1; +} + +int main(int argc, char **argv) +{ + odph_odpthread_t thread_tbl[MAX_NB_WORKER]; + odp_pool_t pool; + odp_pool_param_t params; + odp_instance_t instance; + odph_odpthread_params_t thr_params; + odp_cpumask_t cpumask; + int cpu, i, j, nb_worker; + uint8_t mac[ODPH_ETHADDR_LEN]; + uint8_t *dst_mac; + app_args_t *args; + struct thread_arg_s *thr_arg; + char *oif; + + if (odp_init_global(&instance, NULL, NULL)) { + printf("Error: ODP global init failed.\n"); + exit(1); + } + + if (odp_init_local(instance, ODP_THREAD_CONTROL)) { + printf("Error: ODP local init failed.\n"); + exit(1); + } + + /* Clear global argument and initialize the dest mac as 2:0:0:0:0:x */ + memset(&global, 0, sizeof(global)); + mac[0] = 2; + for (i = 0; i < MAX_NB_PKTIO; i++) { + mac[ODPH_ETHADDR_LEN - 1] = (uint8_t)i; + memcpy(global.eth_dest_mac[i].addr, mac, ODPH_ETHADDR_LEN); + } + + /* Initialize the thread arguments */ + for (i = 0; i < MAX_NB_WORKER; i++) { + thr_arg = &global.worker_args[i]; + for (j = 0; j < MAX_NB_PKTIO; j++) { + thr_arg->thr_idx = INVALID_ID; + thr_arg->pktio[j].txq_idx = INVALID_ID; + memset(thr_arg->pktio[j].rxq, INVALID_ID, + sizeof(thr_arg->pktio[j].rxq)); + } + } + + /* Parse cmdline arguments */ + args = &global.cmd_args; + parse_cmdline_args(argc, argv, args); + + /* Init l3fwd table */ + init_fwd_db(); + + /* Add route into table */ + for (i = 0; i < MAX_NB_ROUTE; i++) { + if (args->route_str[i]) { + create_fwd_db_entry(args->route_str[i], &oif, &dst_mac); + if (oif == NULL) { + printf("Error: fail to create route entry.\n"); + exit(1); + } + + j = find_port_id_by_name(oif, args); + if (j == -1) { + printf("Error: port %s not used.\n", oif); + exit(1); + } + + if (dst_mac) + args->dest_mac_changed[j] = 1; + } + } + + print_info(NO_PATH(argv[0]), args); + + /* Create packet pool */ + odp_pool_param_init(&params); + params.pkt.seg_len = POOL_SEG_LEN; + params.pkt.len = POOL_SEG_LEN; + params.pkt.num = POOL_NUM_PKT; + params.type = ODP_POOL_PACKET; + + pool = odp_pool_create("packet pool", &params); + + if (pool == ODP_POOL_INVALID) { + printf("Error: packet pool create failed.\n"); + exit(1); + } + + /* Resolve fwd db*/ + for (i = 0; i < args->if_count; i++) { + struct l3fwd_pktio_s *port; + char *if_name; + + if_name = args->if_names[i]; + port = &global.l3fwd_pktios[i]; + if (create_pktio(if_name, pool, port)) { + printf("Error: create pktio %s\n", if_name); + exit(1); + } + odp_pktio_mac_addr(port->pktio, mac, ODPH_ETHADDR_LEN); + resolve_fwd_db(if_name, i, mac); + memcpy(port->mac_addr.addr, mac, ODPH_ETHADDR_LEN); + } + setup_fwd_db(); + dump_fwd_db(); + + /* Dicide available workers */ + nb_worker = MAX_NB_WORKER; + if (args->worker_count) + nb_worker = args->worker_count; + nb_worker = odp_cpumask_default_worker(&cpumask, nb_worker); + args->worker_count = nb_worker; + + /* Setup rx and tx queues for each port */ + setup_worker_qconf(args); + print_qconf_table(args); + + /* Decide ip lookup method */ + if (args->hash_mode) + global.fwd_func = l3fwd_pkt_hash; + else + global.fwd_func = l3fwd_pkt_lpm; + + /* Start all the available ports */ + for (i = 0; i < args->if_count; i++) { + struct l3fwd_pktio_s *port; + char *if_name; + char buf[32]; + + if_name = args->if_names[i]; + port = &global.l3fwd_pktios[i]; + /* start pktio */ + if (odp_pktio_start(port->pktio)) { + printf("unable to start pktio: %s\n", if_name); + exit(1); + } + + sprintf(buf, "%02x:%02x:%02x:%02x:%02x:%02x", + port->mac_addr.addr[0], + port->mac_addr.addr[1], + port->mac_addr.addr[2], + port->mac_addr.addr[3], + port->mac_addr.addr[4], + port->mac_addr.addr[5]); + printf("start pktio: %s, mac %s\n", if_name, buf); + } + + odp_barrier_init(&barrier, nb_worker + 1); + + memset(&thr_params, 0, sizeof(thr_params)); + thr_params.start = run_worker; + thr_params.thr_type = ODP_THREAD_WORKER; + thr_params.instance = instance; + + memset(thread_tbl, 0, sizeof(thread_tbl)); + cpu = odp_cpumask_first(&cpumask); + for (i = 0; i < nb_worker; i++) { + struct thread_arg_s *arg; + odp_cpumask_t thr_mask; + + arg = &global.worker_args[i]; + arg->nb_pktio = args->if_count; + odp_cpumask_zero(&thr_mask); + odp_cpumask_set(&thr_mask, cpu); + thr_params.arg = arg; + odph_odpthreads_create(&thread_tbl[i], &thr_mask, + &thr_params); + cpu = odp_cpumask_next(&cpumask, cpu); + } + + print_speed_stats(nb_worker, args->duration, PRINT_INTERVAL); + exit_threads = 1; + + /* wait for other threads to join */ + for (i = 0; i < nb_worker; i++) + odph_odpthreads_join(&thread_tbl[i]); + + /* release resource on exit */ + destroy_fwd_db(); + + /* if_names share a single buffer, so only one free */ + free(args->if_names[0]); + + for (i = 0; i < MAX_NB_ROUTE; i++) + free(args->route_str[i]); + + if (odp_pool_destroy(pool)) { + printf("Error: pool destroy\n"); + exit(EXIT_FAILURE); + } + + if (odp_term_local()) { + printf("Error: term local\n"); + exit(EXIT_FAILURE); + } + + if (odp_term_global(instance)) { + printf("Error: term global\n"); + exit(EXIT_FAILURE); + } + + return 0; +} diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c new file mode 100644 index 0000000..b3a237f --- /dev/null +++ b/example/l3fwd/odp_l3fwd_db.c @@ -0,0 +1,432 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +#include <stdlib.h> +#include <string.h> + +#include <example_debug.h> +#include <odp_api.h> +#include <odp_l3fwd_db.h> + +/** Jenkins hash support. + * + * Copyright (C) 2006 Bob Jenkins (bob_jenkins(a)burtleburtle.net) + * + * http://burtleburtle.net/bob/hash/ + * + * These are the credits from Bob's sources: + * + * lookup3.c, by Bob Jenkins, May 2006, Public Domain. + * + * These are functions for producing 32-bit hashes for hash table lookup. + * hashword(), hashlittle(), hashlittle2(), hashbig(), mix(), and final() + * are externally useful functions. Routines to test the hash are included + * if SELF_TEST is defined. You can use this free for any purpose. It's in + * the public domain. It has no warranty. + * + * $FreeBSD$ + */ +#define JHASH_GOLDEN_RATIO 0x9e3779b9 +#define rot(x, k) (((x) << (k)) | ((x) >> (32 - (k)))) +#define FWD_BJ3_MIX(a, b, c) \ +{ \ + a -= c; a ^= rot(c, 4); c += b; \ + b -= a; b ^= rot(a, 6); a += c; \ + c -= b; c ^= rot(b, 8); b += a; \ + a -= c; a ^= rot(c, 16); c += b; \ + b -= a; b ^= rot(a, 19); a += c; \ + c -= b; c ^= rot(b, 4); b += a; \ +} + +/** + * Compute hash value from a flow + */ +static inline +uint64_t l3fwd_calc_hash(ipv4_tuple5_t *key) +{ + uint64_t l4_ports = 0; + uint32_t dst_ip, src_ip; + + src_ip = key->src_ip; + dst_ip = key->dst_ip + JHASH_GOLDEN_RATIO; + FWD_BJ3_MIX(src_ip, dst_ip, l4_ports); + + return l4_ports; +} + +/** + * Parse text string representing an IPv4 address or subnet + * + * String is of the format "XXX.XXX.XXX.XXX(/W)" where + * "XXX" is decimal value and "/W" is optional subnet length + * + * @param ipaddress Pointer to IP address/subnet string to convert + * @param addr Pointer to return IPv4 address, host endianness + * @param depth Pointer to subnet bit width + * @return 0 if successful else -1 + */ +static inline +int parse_ipv4_string(char *ipaddress, uint32_t *addr, uint32_t *depth) +{ + int b[4]; + int qualifier = 32; + int converted; + uint32_t addr_le; + + if (strchr(ipaddress, '/')) { + converted = sscanf(ipaddress, "%d.%d.%d.%d/%d", + &b[3], &b[2], &b[1], &b[0], + &qualifier); + if (5 != converted) + return -1; + } else { + converted = sscanf(ipaddress, "%d.%d.%d.%d", + &b[3], &b[2], &b[1], &b[0]); + if (4 != converted) + return -1; + } + + if ((b[0] > 255) || (b[1] > 255) || (b[2] > 255) || (b[3] > 255)) + return -1; + if (!qualifier || (qualifier > 32)) + return -1; + + addr_le = b[0] | b[1] << 8 | b[2] << 16 | b[3] << 24; + *addr = odp_le_to_cpu_32(addr_le); + *depth = qualifier; + + return 0; +} + +/** + * Generate text string representing IPv4 range/subnet, output + * in "XXX.XXX.XXX.XXX/W" format + * + * @param b Pointer to buffer to store string + * @param range Pointer to IPv4 address range + * + * @return Pointer to supplied buffer + */ +static inline +char *ipv4_subnet_str(char *b, ip_addr_range_t *range) +{ + sprintf(b, "%d.%d.%d.%d/%d", + 0xFF & ((range->addr) >> 24), + 0xFF & ((range->addr) >> 16), + 0xFF & ((range->addr) >> 8), + 0xFF & ((range->addr) >> 0), + range->depth); + return b; +} + +/** + * Generate text string representing MAC address + * + * @param b Pointer to buffer to store string + * @param mac Pointer to MAC address + * + * @return Pointer to supplied buffer + */ +static inline +char *mac_addr_str(char *b, odph_ethaddr_t *mac) +{ + uint8_t *byte; + + byte = mac->addr; + sprintf(b, "%02X:%02X:%02X:%02X:%02X:%02X", + byte[0], byte[1], byte[2], byte[3], byte[4], byte[5]); + return b; +} + +/** + * Flow cache table entry + */ +typedef struct flow_entry_s { + ipv4_tuple5_t key; /**< match key */ + struct flow_entry_s *next; /**< next entry on the list */ + fwd_db_entry_t *fwd_entry; /**< entry info in db */ +} flow_entry_t; + +/** + * Flow cache table bucket + */ +typedef struct flow_bucket_s { + odp_spinlock_t lock; /**< Bucket lock*/ + flow_entry_t *next; /**< Pointer to first flow entry in bucket*/ +} flow_bucket_t; + +/** + * Flow hash table, fast lookup cache + */ +typedef struct flow_table_s { + flow_bucket_t *bucket; + uint32_t count; +} flow_table_t; + +static flow_table_t fwd_lookup_cache; + +void init_fwd_hash_cache(void) +{ + odp_shm_t hash_shm; + flow_bucket_t *bucket; + uint32_t bucket_count; + uint32_t i; + + bucket_count = FWD_DEF_BUCKET_COUNT; + + /*Reserve memory for Routing hash table*/ + hash_shm = odp_shm_reserve("route_table", + sizeof(flow_bucket_t) * bucket_count, + ODP_CACHE_LINE_SIZE, 0); + + bucket = odp_shm_addr(hash_shm); + if (!bucket) { + EXAMPLE_ERR("Error: shared mem alloc failed.\n"); + exit(-1); + } + + fwd_lookup_cache.bucket = bucket; + fwd_lookup_cache.count = bucket_count; + + /*Initialize Locks*/ + for (i = 0; i < bucket_count; i++) { + bucket = &fwd_lookup_cache.bucket[i]; + odp_spinlock_init(&bucket->lock); + bucket->next = NULL; + } +} + +static inline +int match_key_flow(ipv4_tuple5_t *key, flow_entry_t *flow) +{ + if (key->src_ip == flow->key.src_ip && + key->dst_ip == flow->key.dst_ip && + key->src_port == flow->key.src_port && + key->dst_port == flow->key.dst_port && + key->proto == flow->key.proto) + return 1; + + return 0; +} + +static inline +flow_entry_t *lookup_fwd_cache(ipv4_tuple5_t *key, flow_bucket_t *bucket) +{ + flow_entry_t *rst; + + for (rst = bucket->next; rst != NULL; rst = rst->next) { + if (match_key_flow(key, rst)) + break; + } + + return rst; +} + +static inline +flow_entry_t *insert_fwd_cache(ipv4_tuple5_t *key, + flow_bucket_t *bucket, + fwd_db_entry_t *entry) +{ + flow_entry_t *flow; + + if (!entry) + return NULL; + + flow = malloc(sizeof(flow_entry_t)); + flow->key = *key; + flow->fwd_entry = entry; + + odp_spinlock_lock(&bucket->lock); + if (!bucket->next) { + bucket->next = flow; + } else { + flow->next = bucket->next; + bucket->next = flow; + } + odp_spinlock_unlock(&bucket->lock); + + return flow; +} + +/** Global pointer to fwd db */ +fwd_db_t *fwd_db; + +void init_fwd_db(void) +{ + odp_shm_t shm; + + shm = odp_shm_reserve("shm_fwd_db", + sizeof(fwd_db_t), + ODP_CACHE_LINE_SIZE, + 0); + + fwd_db = odp_shm_addr(shm); + + if (fwd_db == NULL) { + EXAMPLE_ERR("Error: shared mem alloc failed.\n"); + exit(EXIT_FAILURE); + } + memset(fwd_db, 0, sizeof(*fwd_db)); +} + +int create_fwd_db_entry(char *input, char **oif, uint8_t **dst_mac) +{ + int pos = 0; + char *local; + char *str; + char *save; + char *token; + fwd_db_entry_t *entry = &fwd_db->array[fwd_db->index]; + + *oif = NULL; + *dst_mac = NULL; + + /* Verify we haven't run out of space */ + if (MAX_DB <= fwd_db->index) + return -1; + + /* Make a local copy */ + local = malloc(strlen(input) + 1); + if (NULL == local) + return -1; + strcpy(local, input); + + /* Setup for using "strtok_r" to search input string */ + str = local; + save = NULL; + + /* Parse tokens separated by ',' */ + while (NULL != (token = strtok_r(str, ",", &save))) { + str = NULL; /* reset str for subsequent strtok_r calls */ + + /* Parse token based on its position */ + switch (pos) { + case 0: + parse_ipv4_string(token, + &entry->subnet.addr, + &entry->subnet.depth); + break; + case 1: + strncpy(entry->oif, token, OIF_LEN - 1); + entry->oif[OIF_LEN - 1] = 0; + *oif = entry->oif; + break; + case 2: + odph_eth_addr_parse(&entry->dst_mac, token); + *dst_mac = entry->dst_mac.addr; + break; + + default: + printf("ERROR: extra token \"%s\" at position %d\n", + token, pos); + break; + } + + /* Advance to next position */ + pos++; + } + + /* Add route to the list */ + fwd_db->index++; + entry->next = fwd_db->list; + fwd_db->list = entry; + + free(local); + return 0; +} + +void resolve_fwd_db(char *intf, int portid, uint8_t *mac) +{ + fwd_db_entry_t *entry; + + /* Walk the list and attempt to set output and MAC */ + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { + if (strcmp(intf, entry->oif)) + continue; + + entry->oif_id = portid; + memcpy(entry->src_mac.addr, mac, ODPH_ETHADDR_LEN); + } +} + +void dump_fwd_db_entry(fwd_db_entry_t *entry) +{ + char subnet_str[MAX_STRING]; + char mac_str[MAX_STRING]; + + mac_addr_str(mac_str, &entry->dst_mac); + printf("%-32s%-32s%-16s\n", + ipv4_subnet_str(subnet_str, &entry->subnet), + entry->oif, mac_str); +} + +void dump_fwd_db(void) +{ + fwd_db_entry_t *entry; + + printf("Routing table\n" + "-----------------\n" + "%-32s%-32s%-16s\n", + "subnet", "next_hop", "dest_mac"); + + for (entry = fwd_db->list; NULL != entry; entry = entry->next) + dump_fwd_db_entry(entry); + + printf("\n"); +} + +void destroy_fwd_db(void) +{ + flow_bucket_t *bucket; + flow_entry_t *flow, *tmp; + uint32_t i; + + for (i = 0; i < fwd_lookup_cache.count; i++) { + bucket = &fwd_lookup_cache.bucket[i]; + flow = bucket->next; + odp_spinlock_lock(&bucket->lock); + while (flow) { + tmp = flow->next; + free(flow); + flow = tmp; + } + odp_spinlock_unlock(&bucket->lock); + } +} + +fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key) +{ + fwd_db_entry_t *entry; + flow_entry_t *flow; + flow_bucket_t *bucket; + uint64_t hash; + + /* first find in cache */ + hash = l3fwd_calc_hash(key); + hash &= fwd_lookup_cache.count - 1; + bucket = &fwd_lookup_cache.bucket[hash]; + flow = lookup_fwd_cache(key, bucket); + if (flow) + return flow->fwd_entry; + + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { + uint32_t mask; + + mask = ((1u << entry->subnet.depth) - 1) << + (32 - entry->subnet.depth); + + if (entry->subnet.addr == (key->dst_ip & mask)) + break; + } + + insert_fwd_cache(key, bucket, entry); + + return entry; +} diff --git a/example/l3fwd/odp_l3fwd_db.h b/example/l3fwd/odp_l3fwd_db.h new file mode 100644 index 0000000..d9f6d61 --- /dev/null +++ b/example/l3fwd/odp_l3fwd_db.h @@ -0,0 +1,136 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ODP_L3FWD_DB_H_ +#define _ODP_L3FWD_DB_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp_api.h> +#include <odp/helper/eth.h> + +#define OIF_LEN 32 +#define MAX_DB 32 +#define MAX_STRING 32 + +/** + * Default number of flows + */ +#define FWD_DEF_FLOW_COUNT 100000 + +/** + * Default Hash bucket number + */ +#define FWD_DEF_BUCKET_COUNT (FWD_DEF_FLOW_COUNT / 8) + +/** + * IP address range (subnet) + */ +typedef struct ip_addr_range_s { + uint32_t addr; /**< IP address, host endianness */ + uint32_t depth; /**< subnet bit width */ +} ip_addr_range_t; + +/** + * TCP/UDP flow + */ +typedef struct ipv4_tuple5_s { + uint32_t src_ip; + uint32_t dst_ip; + uint16_t src_port; + uint16_t dst_port; + uint8_t proto; +} ipv4_tuple5_t; + +/** + * Forwarding data base entry + */ +typedef struct fwd_db_entry_s { + struct fwd_db_entry_s *next; /**< Next entry on list */ + char oif[OIF_LEN]; /**< Output interface name */ + int oif_id; /**< Output interface idx */ + odph_ethaddr_t src_mac; /**< Output source MAC */ + odph_ethaddr_t dst_mac; /**< Output destination MAC */ + ip_addr_range_t subnet; /**< Subnet for this router */ +} fwd_db_entry_t; + +/** + * Forwarding data base + */ +typedef struct fwd_db_s { + uint32_t index; /**< Next available entry */ + fwd_db_entry_t *list; /**< List of active routes */ + fwd_db_entry_t array[MAX_DB]; /**< Entry storage */ +} fwd_db_t; + +/** Global pointer to fwd db */ +extern fwd_db_t *fwd_db; + +/** + * Initialize FWD DB + */ +void init_fwd_db(void); + +/** + * Initialize forward lookup cache based on hash + */ +void init_fwd_hash_cache(void); + +/** + * Create a forwarding database entry + * + * String is of the format "SubNet,Intf,NextHopMAC" + * + * @param input Pointer to string describing route + * @param oif Pointer to out interface name, as a return value + * @param dst_mac Pointer to dest mac for output packet, as a return value + * + * @return 0 if successful else -1 + */ +int create_fwd_db_entry(char *input, char **oif, uint8_t **dst_mac); + +/** + * Scan FWD DB entries and resolve output queue and source MAC address + * + * @param intf Interface name string + * @param portid Output queue for packet transmit + * @param mac MAC address of this interface + */ +void resolve_fwd_db(char *intf, int portid, uint8_t *mac); + +/** + * Display one forwarding database entry + * + * @param entry Pointer to entry to display + */ +void dump_fwd_db_entry(fwd_db_entry_t *entry); + +/** + * Display the forwarding database + */ +void dump_fwd_db(void); + +/** + * Destroy the forwarding database + */ +void destroy_fwd_db(void); + +/** + * Find a matching forwarding database entry + * + * @param key ipv4 tuple + * + * @return pointer to forwarding DB entry else NULL + */ +fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/example/l3fwd/odp_l3fwd_lpm.c b/example/l3fwd/odp_l3fwd_lpm.c new file mode 100644 index 0000000..65a3389 --- /dev/null +++ b/example/l3fwd/odp_l3fwd_lpm.c @@ -0,0 +1,224 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +#include <stdio.h> +#include <stdlib.h> + +#include <example_debug.h> +#include <odp_api.h> + +#include <odp_l3fwd_lpm.h> + +/** + * This is a simple implementation of lpm based on patricia tree. + * + * Tradeoff exists between memory consumption and lookup time. + * Currently it prefers 5 levels: {16, 4, 4, 4, 4}, could be 3 + * levels: {16, 8, 8} by defining FIB_NEXT_STRIDE as 8. Other + * levels are also possible. + * + * the ip here is host endian, when doing init or lookup, the + * caller should do endianness conversion if needed. + */ + +#define FIB_IP_WIDTH 32 +#define FIB_FIRST_STRIDE 16 +#define FIB_NEXT_STRIDE 4 +#define FIB_NEXT_SIZE (1 << FIB_NEXT_STRIDE) +#define FIB_SUB_COUNT 16384 +#define DEPTH_TO_MASK(depth) ((1 << (depth)) - 1) + +typedef struct fib_node_s { + union { + uint32_t next_hop; + struct fib_node_s *next; /* next level table */ + }; + uint8_t valid :1; /* 1, this node has a valid next hop */ + uint8_t end :1; /* 0, next points to the extended table */ + uint8_t depth :6; /* bit length of subnet mask */ +} fib_node_t; + +typedef struct fib_sub_tbl_s { + fib_node_t *fib_nodes; + uint32_t fib_count; + uint32_t fib_idx; +} fib_sub_tbl_t; + +static fib_node_t fib_rt_tbl[1 << FIB_FIRST_STRIDE]; +static fib_sub_tbl_t fib_lpm_cache; + +static inline fib_node_t *fib_alloc_sub(void) +{ + fib_node_t *sub_tbl = NULL; + uint32_t i, nb_entry; + + /* extend to next level */ + if (fib_lpm_cache.fib_idx < fib_lpm_cache.fib_count) { + nb_entry = (fib_lpm_cache.fib_idx + 1) * FIB_NEXT_SIZE; + sub_tbl = &fib_lpm_cache.fib_nodes[nb_entry]; + fib_lpm_cache.fib_idx++; + for (i = 0; i < nb_entry; i++) { + sub_tbl[i].valid = 0; + sub_tbl[i].end = 1; + } + } + + return sub_tbl; +} + +static void fib_update_node(fib_node_t *fe, int port, int depth) +{ + fib_node_t *p; + int i; + + if (fe->end) { + if (!fe->valid) { + fe->depth = depth; + fe->next_hop = port; + fe->valid = 1; + } else if (fe->depth <= depth) { + fe->next_hop = port; + fe->depth = depth; + } + + return; + } + + for (i = 0; i < FIB_NEXT_SIZE; i++) { + p = &fe->next[i]; + if (p->end) + fib_update_node(p, port, depth); + } +} + +static void fib_insert_node(fib_node_t *fe, uint32_t ip, uint32_t next_hop, + int ip_width, int eat_bits, int depth) +{ + int i; + uint32_t idx, port; + fib_node_t *p; + + if (fe->end) { + port = fe->next_hop; + p = fib_alloc_sub(); + if (!p) + return; + + fe->next = p; + fe->end = 0; + if (fe->valid) { + for (i = 0; i < FIB_NEXT_SIZE; i++) { + p = &fe->next[i]; + p->next_hop = port; + p->depth = fe->depth; + } + } + } + if (depth - eat_bits <= FIB_NEXT_STRIDE) { + ip_width -= depth - eat_bits; + idx = ip >> ip_width; + ip &= DEPTH_TO_MASK(ip_width); + p = &fe->next[idx]; + fib_update_node(p, next_hop, depth); + } else { + ip_width -= FIB_NEXT_STRIDE; + idx = ip >> ip_width; + p = &fe->next[idx]; + ip &= DEPTH_TO_MASK(ip_width); + eat_bits += FIB_NEXT_STRIDE; + fib_insert_node(p, ip, next_hop, ip_width, eat_bits, depth); + } +} + +void fib_tbl_init(void) +{ + int i; + fib_node_t *fe; + uint32_t size; + odp_shm_t lpm_shm; + + for (i = 0; i < (1 << FIB_FIRST_STRIDE); i++) { + fe = &fib_rt_tbl[i]; + fe->valid = 0; + fe->end = 1; + fe->depth = 0; + fe->next_hop = 0; + } + + size = FIB_NEXT_SIZE * FIB_SUB_COUNT; + /*Reserve memory for Routing hash table*/ + lpm_shm = odp_shm_reserve("fib_lpm_sub", size, ODP_CACHE_LINE_SIZE, 0); + fe = odp_shm_addr(lpm_shm); + if (!fe) { + EXAMPLE_ERR("Error: shared mem alloc failed for lpm cache.\n"); + exit(-1); + } + + fib_lpm_cache.fib_nodes = fe; + fib_lpm_cache.fib_count = FIB_SUB_COUNT; + fib_lpm_cache.fib_idx = 0; +} + +void fib_tbl_insert(uint32_t ip, int port, int depth) +{ + fib_node_t *fe, *p; + uint32_t idx; + int i, j; + int nb_bits; + + nb_bits = FIB_FIRST_STRIDE; + idx = ip >> (FIB_IP_WIDTH - nb_bits); + fe = &fib_rt_tbl[idx]; + if (depth <= nb_bits) { + if (fe->end) { + fe->next_hop = port; + fe->depth = depth; + fe->valid = 1; + return; + } + + for (i = 0; i < FIB_NEXT_SIZE; i++) { + p = &fe->next[i]; + if (p->end) + fib_update_node(p, port, depth); + else + for (j = 0; j < FIB_NEXT_SIZE; j++) + fib_update_node(&p->next[j], port, + depth); + } + + return; + } + + /* need to check sub table */ + ip &= DEPTH_TO_MASK(FIB_IP_WIDTH - nb_bits); + fib_insert_node(fe, ip, port, FIB_IP_WIDTH - nb_bits, nb_bits, depth); +} + +int fib_tbl_lookup(uint32_t ip, int *port) +{ + fib_node_t *fe; + uint32_t idx; + int nb_bits; + + nb_bits = FIB_IP_WIDTH - FIB_FIRST_STRIDE; + idx = ip >> nb_bits; + fe = &fib_rt_tbl[idx]; + + ip &= DEPTH_TO_MASK(nb_bits); + while (!fe->end) { + nb_bits -= FIB_NEXT_STRIDE; + idx = ip >> nb_bits; + fe = &fe->next[idx]; + ip &= DEPTH_TO_MASK(nb_bits); + } + *port = fe->next_hop; + + return fe->valid ? 0 : -1; +} diff --git a/example/l3fwd/odp_l3fwd_lpm.h b/example/l3fwd/odp_l3fwd_lpm.h new file mode 100644 index 0000000..9256642 --- /dev/null +++ b/example/l3fwd/odp_l3fwd_lpm.h @@ -0,0 +1,20 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ODP_L3FWD_LPM_H_ +#define _ODP_L3FWD_LPM_H_ + +#ifdef __cplusplus +extern "C" { +#endif +void fib_tbl_init(void); +void fib_tbl_insert(uint32_t ip, int port, int depth); +int fib_tbl_lookup(uint32_t ip, int *port); +#ifdef __cplusplus +} +#endif + +#endif diff --git a/example/m4/configure.m4 b/example/m4/configure.m4 index bbda38f..620db04 100644 --- a/example/m4/configure.m4 +++ b/example/m4/configure.m4 @@ -12,12 +12,13 @@ AC_ARG_ENABLE([test-example], AC_CONFIG_FILES([example/classifier/Makefile example/generator/Makefile + example/hello/Makefile example/ipsec/Makefile - example/Makefile + example/l2fwd_simple/Makefile + example/l3fwd/Makefile example/packet/Makefile + example/switch/Makefile example/time/Makefile example/timer/Makefile example/traffic_mgmt/Makefile - example/l2fwd_simple/Makefile - example/switch/Makefile - example/hello/Makefile]) + example/Makefile]) ----------------------------------------------------------------------- Summary of changes: example/Makefile.am | 12 +- example/l3fwd/.gitignore | 1 + example/l3fwd/Makefile.am | 20 + example/l3fwd/odp_l3fwd.c | 1086 ++++++++++++++++++++ example/l3fwd/odp_l3fwd_db.c | 494 +++++++++ example/l3fwd/odp_l3fwd_db.h | 141 +++ example/l3fwd/odp_l3fwd_lpm.c | 224 ++++ .../cpu_arch.h => example/l3fwd/odp_l3fwd_lpm.h | 14 +- example/l3fwd/odp_l3fwd_run.sh | 28 + example/{l2fwd_simple => l3fwd}/udp64.pcap | Bin example/m4/configure.m4 | 9 +- 11 files changed, 2015 insertions(+), 14 deletions(-) create mode 100644 example/l3fwd/.gitignore create mode 100644 example/l3fwd/Makefile.am create mode 100644 example/l3fwd/odp_l3fwd.c create mode 100644 example/l3fwd/odp_l3fwd_db.c create mode 100644 example/l3fwd/odp_l3fwd_db.h create mode 100644 example/l3fwd/odp_l3fwd_lpm.c copy platform/linux-generic/arch/powerpc/odp/api/cpu_arch.h => example/l3fwd/odp_l3fwd_lpm.h (52%) create mode 100755 example/l3fwd/odp_l3fwd_run.sh copy example/{l2fwd_simple => l3fwd}/udp64.pcap (100%) hooks/post-receive --

10 years

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

git-lng-odp